pbackup12345 opened a new issue, #547:
URL: https://github.com/apache/solr-operator/issues/547
I have a GKE autoPilot pod with a relatively vanilla setup with
tlsTermination at Ingress and 3 pods. Everything works except for the backup
feature.
Following are the relevant parts of my setup:
Main yaml on S3 location:
```
backupRepositories:
- name: "gcs-backups-1"
gcs:
bucket: "backupbx"
gcsCredentialSecret:
name: "gcssecret1"
key: "service-account-key.json"
baseLocation: "d"
- name: "s3-backups-1"
s3:
region: "us-east-1" # Required
bucket: "solr-xsearch" # Required
credentials:
accessKeyIdSecret: # Optional
name: aws-secrets
key: access-key-id
secretAccessKeySecret: # Optional
name: aws-secrets
key: secret-access-key
```
My backup yaml:
```
apiVersion: solr.apache.org/v1beta1
kind: SolrBackup
metadata:
name: local-backup14
namespace: sop030
spec:
repositoryName: "s3-backups-1"
solrCloud: explore
collections:
- dsearch
```
The backup actually starts and both location (S3 and GCS) receives files as
well, but after a while the backup process stops.
There're no solr error messages, but this is the relevant portion of the
logs of the pod which does the backup:
```
2023-04-08 14:42:00.734 INFO
(parallelCoreAdminExecutor-19-thread-3-processing-n:explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr
x:dsearch_shard1_replica_n1 local-backup14-dsearch8519612781285 BACKUPCORE)
[c:dsearch s:shard1 r:core_node2 x:dsearch_shard1_replica_n1]
o.a.s.h.IncrementalShardBackup Done creating backup snapshot at
s3:/local-backup14-dsearch/dsearch/
shardBackupMetadataFile:org.apache.solr.core.backup.ShardBackupId@46768e0a
2023-04-08 14:42:01.396 INFO (qtp1306246648-19) [ ]
o.a.s.h.a.CoreAdminOperation Checking request status for :
local-backup14-dsearch8519612781285
2023-04-08 14:42:01.396 INFO (qtp1306246648-19) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&requestid=local-backup14-dsearch8519612781285&action=REQUESTSTATUS&wt=javabin&version=2}
status=0 QTime=0
2023-04-08 14:42:01.398 INFO
(OverseerThreadFactory-29-thread-5-processing-n:explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr)
[c:dsearch ] o.a.s.c.a.c.BackupCmd Starting to backup ZK data for
backupName=local-backup14-dsearch
2023-04-08 14:42:04.358 INFO (qtp1306246648-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=4
2023-04-08 14:42:04.358 INFO (qtp1306246648-21) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=4
2023-04-08 14:42:09.357 INFO (qtp1306246648-20) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=3
2023-04-08 14:42:14.358 INFO (qtp1306246648-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=3
2023-04-08 14:42:14.361 INFO (qtp1306246648-21) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=5
2023-04-08 14:42:16.585 INFO
(OverseerThreadFactory-29-thread-5-processing-n:explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr)
[c:dsearch ] o.a.s.c.a.c.BackupCmd Completed backing up ZK data for
backupName=local-backup14-dsearch
2023-04-08 14:42:16.655 INFO
(OverseerThreadFactory-29-thread-5-processing-n:explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr)
[c:dsearch ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000043 doesn't exist. Requestor may
have disconnected from ZooKeeper
2023-04-08 14:42:18.400 INFO
(OverseerCollectionConfigSetProcessor-144115198735482887-explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr-n_0000000002)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000043 doesn't exist. Requestor may
have disconnected from ZooKeeper
2023-04-08 14:42:19.358 INFO (qtp1306246648-20) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=3
2023-04-08 14:42:24.359 INFO (qtp1306246648-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=4
2023-04-08 14:42:24.360 INFO (qtp1306246648-22) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=3
2023-04-08 14:42:29.359 INFO (qtp1306246648-20) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=4
2023-04-08 14:42:34.360 INFO (qtp1306246648-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=6
2023-04-08 14:42:34.360 INFO (qtp1306246648-24) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=6
2023-04-08 14:42:39.358 INFO (qtp1306246648-21) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/system params={} status=0 QTime=4
2023-04-08 14:42:42.779 INFO (qtp1306246648-18) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={requestid=local-backup14-dsearch&action=REQUESTSTATUS&wt=json} status=0
QTime=3
2023-04-08 14:42:42.797 INFO (qtp1306246648-22) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={requestid=local-backup14-dsearch&action=REQUESTSTATUS&wt=json} status=0
QTime=2
```
And this is where it seems to die:
```
2023-04-08 14:42:18.400 INFO
(OverseerCollectionConfigSetProcessor-144115198735482887-explore-solrcloud-1.explore-solrcloud-headless.sop030:8983_solr-n_0000000002)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000043 doesn't exist. Requestor may
have disconnected from ZooKeeper
```
From there the log only has endless calls to check the requeststatus with no
results:
```
2023-04-08 14:43:27.884 INFO (qtp1306246648-23) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={requestid=local-backup14-dsearch&action=REQUESTSTATUS&wt=json} status=0
QTime=2
```
Additionally through normal api calls the backup and restore function works
perfectly. Specifically the following runs without a hitch:
```
https://<solr-server>/solr/admin/collections?action=BACKUP&name=local-backup15&collection=dsearch&repository=s3-backups-1&location=/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
