[jira] [Commented] (SOLR-12842) CDCR stops replication and goes into infinite loop of retrying, if one of the updates are corrupted.
[
https://issues.apache.org/jira/browse/SOLR-12842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736137#comment-16736137
]
Webster Homer commented on SOLR-12842:
--
I see this issue a lot, it is a major maintenance headache. It seems that
network issues can create corruption and when it does the cdcr replication for
that collection is stopped until we delete a bad tlog.
I guess we see this several times a month with multiple collections
> CDCR stops replication and goes into infinite loop of retrying, if one of the
> updates are corrupted.
>
>
> Key: SOLR-12842
> URL: https://issues.apache.org/jira/browse/SOLR-12842
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: master (9.0)
>Reporter: Amrit Sarkar
>Priority: Major
>
> Currently, CDCR reads updates from the transaction logs, create UpdateRequest
> and forwards it to the target. If the UpdateRequest sent fails, the same
> request is retried indefinitely until a successful acknowledgment is
> received.
> {code}
> 2018-10-03 00:05:38.878 WARN (cdcr-replicator-35-thread-4) [ ]
> o.a.s.h.CdcrReplicator Failed to forward update request to target: DEMO_DR
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://10.202.153.32:3987/solr/DEMO_DR: version conflict for
> ec53e70d-72fb-42bd-827e-a3fc54e33bad expected=1613259487692455936 actual=-1
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> org.apache.solr.handler.CdcrReplicator.sendRequest(CdcrReplicator.java:140)
> ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:13]
> at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:120)
> ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:13]
> at
> org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
> ~[solr-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:13]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> ~[solr-solrj-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc -
> jpountz - 2018-06-18 16:55:14]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0-zing_18.07.1.0]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0-zing_18.07.1.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0-zing_18.07.1.0]
> {code}
> This design is in place when connected to target is broken and you need to
> halt the forwarding, but better error handling sh
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391544#comment-16391544
]
Webster Homer commented on SOLR-12057:
--
My understanding is that if you have PULL replicas you will also have at least
one NRT or TLOG replica type in the cloud as well. PULL replicas don't do their
own indexing. I wouldn't expect that PULL replicas would need deal with cdcr,
instead they would get any updates from cdcr via their master replica e.g. the
NRT or TLOG replica
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Priority: Major
> Attachments: SOLR-12057.patch, SOLR-12057.patch,
> cdcr-fail-with-tlog-pull.patch, cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388058#comment-16388058
]
Webster Homer edited comment on SOLR-12057 at 3/6/18 4:35 PM:
--
Our setup would likely have either Tlog or NRT replicas. Any setup would have
at least one of those, I wouldn't expect that you'd want to send to a PULL
replica since they cannot be masters.
We would like to favor TLOG replicas since they always give consistent search
results while NRT do not
was (Author: webhomer):
Our setup would likely have either Tlog or NRT replicas. Any setup would have
at least one of those, I wouldn't expect that you'd want to send to a PULL
replica since they cannot be masters.
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Priority: Major
> Attachments: cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388058#comment-16388058
]
Webster Homer commented on SOLR-12057:
--
Our setup would likely have either Tlog or NRT replicas. Any setup would have
at least one of those, I wouldn't expect that you'd want to send to a PULL
replica since they cannot be masters.
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Priority: Major
> Attachments: cdcr-fail-with-tlog-pull.patch
>
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
[
https://issues.apache.org/jira/browse/SOLR-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386822#comment-16386822
]
Webster Homer commented on SOLR-12057:
--
I noticed that the cdcr action=queues returns different results for the target
clouds.
{"responseHeader": {"status": 0,"QTime": 0},"queues": [],"tlogTotalSize":
0,"tlogTotalCount": 0,"updateLogSynchronizer": "stopped"}
and the other
{"responseHeader": {"status": 0,"QTime": 0},"queues": [],"tlogTotalSize":
22254206389,"tlogTotalCount": 2,"updateLogSynchronizer": "started"}
The source is as follows:
{
"responseHeader": {
"status": 0,
"QTime": 5
},
"queues": [
"xxx-mzk01.sial.com:2181,xxx-mzk02.sial.com:2181,xxx-mzk03.sial.com:2181/solr",
[
"b2b-catalog-material-180124T",
[
"queueSize",
0,
"lastTimestamp",
"2018-02-28T18:34:39.704Z"
]
],
"yyy-mzk01.sial.com:2181,yyy-mzk02.sial.com:2181,yyy-mzk03.sial.com:2181/solr",
[
"b2b-catalog-material-180124T",
[
"queueSize",
0,
"lastTimestamp",
"2018-02-28T18:34:39.704Z"
]
]
],
"tlogTotalSize": 1970848,
"tlogTotalCount": 1,
"updateLogSynchronizer": "stopped"
}
> CDCR does not replicate to Collections with TLOG Replicas
> -
>
> Key: SOLR-12057
> URL: https://issues.apache.org/jira/browse/SOLR-12057
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.2
>Reporter: Webster Homer
>Priority: Major
>
> We created a collection using TLOG replicas in our QA clouds.
> We have a locally hosted solrcloud with 2 nodes, all our collections have 2
> shards. We use CDCR to replicate the collections from this environment to 2
> data centers hosted in Google cloud. This seems to work fairly well for our
> collections with NRT replicas. However the new TLOG collection has problems.
>
> The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
> shards per collection with 2 replicas per shard.
>
> We never see data show up in the cloud collections, but we do see tlog files
> show up on the cloud servers. I can see that all of the servers have cdcr
> started, buffers are disabled.
> The cdcr source configuration is:
>
> "requestHandler":{"/cdcr":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica":[
> {
>
> "zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"},
> {
>
> "zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
> "source":"b2b-catalog-material-180124T",
> "target":"b2b-catalog-material-180124T"}],
> "replicator":{
> "threadPoolSize":4,
> "schedule":500,
> "batchSize":250},
> "updateLogSynchronizer":\{"schedule":6
>
> The target configurations in the 2 clouds are the same:
> "requestHandler":{"/cdcr":{ "name":"/cdcr",
> "class":"solr.CdcrRequestHandler", "buffer":{"defaultState":"disabled"}}}
>
> All of our collections have a timestamp field, index_date. In the source
> collection all the records have a date of 2/28/2018 but the target
> collections have a latest date of 1/26/2018
>
> I don't see cdcr errors in the logs, but we use logstash to search them, and
> we're still perfecting that.
>
> We have a number of similar collections that behave correctly. This is the
> only collection that is a TLOG collection. It appears that CDCR doesn't
> support TLOG collections.
>
> It looks like the data is getting to the target servers. I see tlog files
> with the right timestamps. Looking at the timestamps on the documents in the
> collection none of the data appears to have been loaded.In the solr.log I see
> lots of /cdcr messages action=LASTPROCESSEDVERSION,
> action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
>
> no errors
>
> Target collections autoCommit is set to 6 I tried sending a commit
> explicitly no difference. cdcr is uploading data, but no new data appears in
> the collection.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12057) CDCR does not replicate to Collections with TLOG Replicas
Webster Homer created SOLR-12057:
Summary: CDCR does not replicate to Collections with TLOG Replicas
Key: SOLR-12057
URL: https://issues.apache.org/jira/browse/SOLR-12057
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: CDCR
Affects Versions: 7.2
Reporter: Webster Homer
We created a collection using TLOG replicas in our QA clouds.
We have a locally hosted solrcloud with 2 nodes, all our collections have 2
shards. We use CDCR to replicate the collections from this environment to 2
data centers hosted in Google cloud. This seems to work fairly well for our
collections with NRT replicas. However the new TLOG collection has problems.
The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
shards per collection with 2 replicas per shard.
We never see data show up in the cloud collections, but we do see tlog files
show up on the cloud servers. I can see that all of the servers have cdcr
started, buffers are disabled.
The cdcr source configuration is:
"requestHandler":{"/cdcr":{
"name":"/cdcr",
"class":"solr.CdcrRequestHandler",
"replica":[
{
"zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
"source":"b2b-catalog-material-180124T",
"target":"b2b-catalog-material-180124T"},
{
"zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
"source":"b2b-catalog-material-180124T",
"target":"b2b-catalog-material-180124T"}],
"replicator":{
"threadPoolSize":4,
"schedule":500,
"batchSize":250},
"updateLogSynchronizer":\{"schedule":6
The target configurations in the 2 clouds are the same:
"requestHandler":{"/cdcr":{ "name":"/cdcr", "class":"solr.CdcrRequestHandler",
"buffer":{"defaultState":"disabled"}}}
All of our collections have a timestamp field, index_date. In the source
collection all the records have a date of 2/28/2018 but the target collections
have a latest date of 1/26/2018
I don't see cdcr errors in the logs, but we use logstash to search them, and
we're still perfecting that.
We have a number of similar collections that behave correctly. This is the only
collection that is a TLOG collection. It appears that CDCR doesn't support TLOG
collections.
It looks like the data is getting to the target servers. I see tlog files with
the right timestamps. Looking at the timestamps on the documents in the
collection none of the data appears to have been loaded.In the solr.log I see
lots of /cdcr messages action=LASTPROCESSEDVERSION,
action=COLLECTIONCHECKPOINT, and action=SHARDCHECKPOINT
no errors
Target collections autoCommit is set to 6 I tried sending a commit
explicitly no difference. cdcr is uploading data, but no new data appears in
the collection.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11724) Cdcr Bootstrapping does not cause "index copying" to follower nodes on Target
[
https://issues.apache.org/jira/browse/SOLR-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384089#comment-16384089
]
Webster Homer commented on SOLR-11724:
--
This may be a different issue, but I see similar behavior with some replicas in
7.2.0
We have a number of replicas which have identical schemas. We found that TLOG
replicas give much more consistent search results.
We created a collection using TLOG replicas in our QA clouds.
We have a locally hosted solrcloud with 2 nodes, all our collections have 2
shards. We use CDCR to replicate the collections from this environment to 2
data centers hosted in Google cloud. This seems to work fairly well for our
collections with NRT replicas. However the new TLOG collection has problems.
The google cloud solrclusters have 4 nodes each (3 separate Zookeepers). 2
shards per collection with 2 replicas per shard.
We never see data show up in the tlog replica cloud collections, but we do see
tlog files show up on the cloud servers. I can see that all of the servers have
cdcr started, buffers are disabled.
The cdcr source configuration is:
"requestHandler":{"/cdcr":{
"name":"/cdcr",
"class":"solr.CdcrRequestHandler",
"replica":[
{
"zkHost":"[xxx-mzk01.sial.com:2181|http://xxx-mzk01.sial.com:2181/],[xxx-mzk02.sial.com:2181|http://xxx-mzk02.sial.com:2181/],[xxx-mzk03.sial.com:2181/solr|http://xxx-mzk03.sial.com:2181/solr]";,
"source":"b2b-catalog-material-180124T",
"target":"b2b-catalog-material-180124T"},
{
"zkHost":"[-mzk01.sial.com:2181|http://-mzk01.sial.com:2181/],[-mzk02.sial.com:2181|http://-mzk02.sial.com:2181/],[-mzk03.sial.com:2181/solr|http://-mzk03.sial.com:2181/solr]";,
"source":"b2b-catalog-material-180124T",
"target":"b2b-catalog-material-180124T"}],
"replicator":{
"threadPoolSize":4,
"schedule":500,
"batchSize":250},
"updateLogSynchronizer":\{"schedule":6
The target configurations in the 2 clouds are the same:
"requestHandler":{"/cdcr":{ "name":"/cdcr", "class":"solr.CdcrRequestHandler",
"buffer":{"defaultState":"disabled"}}}
-rw-r--r-- 1 apache apache 596517718 Feb 28 20:19
tlog.029.1593653529934823424
-rw-r--r-- 1 apache apache 647 Feb 28 22:18
tlog.030.1593670744554864641
[apache@uc1b-ecomqa-msc01 tlog]$ pwd
/var/solr/data/b2b-catalog-material-180124T_shard1_replica_t2/data/tlog
All of our collections have a timestamp field, index_date. In the source
collection all the records have a date of 2/28/2018 but the target collections
have a latest date of 1/26/2018
I don't see cdcr errors in the logs.
We have a number of similar collections that behave correctly. This is the only
collection that is a TLOG collection. It appears that CDCR doesn't support TLOG
collections.
We also see the same behavior in our production solrclouds. The collections
that use NRT replicas replicate fine with CDCR, the collection that uses TLOG
replicas do not. Several of the NRT collections have the same configurations as
the tlog colletion, so that seems to be the only difference between them
> Cdcr Bootstrapping does not cause "index copying" to follower nodes on Target
> -
>
> Key: SOLR-11724
> URL: https://issues.apache.org/jira/browse/SOLR-11724
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 7.1
>Reporter: Amrit Sarkar
>Priority: Major
> Attachments: SOLR-11724.patch, SOLR-11724.patch, SOLR-11724.patch,
> SOLR-11724.patch
>
>
> Please find the discussion on:
> http://lucene.472066.n3.nabble.com/Issue-with-CDCR-bootstrapping-in-Solr-7-1-td4365258.html
> If we index significant documents in to Source, stop indexing and then start
> CDCR; bootstrapping only copies the index to leader node of shards of the
> collection, and followers never receive the documents / index until and
> unless atleast one document is inserted again on source; which propels to
> target and target collection trigger index replication to followers.
> This behavior needs to be addressed in proper manner, either at target
> collection or while bootstrapping.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11724) Cdcr Bootstrapping does not cause "index copying" to follower nodes on Target
[ https://issues.apache.org/jira/browse/SOLR-11724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348769#comment-16348769 ] Webster Homer commented on SOLR-11724: -- We are seeing this behavior in 7.2.0. Sometimes the data shows up in a search, but when I look there are no index segments. It appears to be searching just tlogs. This looks like a critical issue to me. We observed the same behavior when we deleted all the data from a collection. On;y tlogs show up and only half are searchable. The configuration worked with Solr 6.1 but doesn't with Solr 7.2. We don't see any errors. We see logs of calls to /cdcr LASTPROCESSEDVERSION in the logs We don't just see this as a bootstrap issue either, we only see about half the data get replicated. I don't know if tlogs are present on the targets, as I don't have access to those file systems > Cdcr Bootstrapping does not cause "index copying" to follower nodes on Target > - > > Key: SOLR-11724 > URL: https://issues.apache.org/jira/browse/SOLR-11724 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: CDCR >Affects Versions: 7.1 >Reporter: Amrit Sarkar >Priority: Major > Attachments: SOLR-11724.patch, SOLR-11724.patch > > > Please find the discussion on: > http://lucene.472066.n3.nabble.com/Issue-with-CDCR-bootstrapping-in-Solr-7-1-td4365258.html > If we index significant documents in to Source, stop indexing and then start > CDCR; bootstrapping only copies the index to leader node of shards of the > collection, and followers never receive the documents / index until and > unless atleast one document is inserted again on source; which propels to > target and target collection trigger index replication to followers. > This behavior needs to be addressed in proper manner, either at target > collection or while bootstrapping. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9175) classes referenced in schema.xml should also support loading from the blob store
[ https://issues.apache.org/jira/browse/SOLR-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122387#comment-16122387 ] Webster Homer commented on SOLR-9175: - This is a significant limitation on the BLOB API. It should be explicitly documented! All of my needs for adding custom code to Solr are either overriding similarity or specialized filter factories! Please update the documentation on the blob store api to reflect this severe limitation! I wasted several hours on it. > classes referenced in schema.xml should also support loading from the blob > store > > > Key: SOLR-9175 > URL: https://issues.apache.org/jira/browse/SOLR-9175 > Project: Solr > Issue Type: Improvement > Components: blobstore >Affects Versions: 5.4.1 >Reporter: King Rhoton > > It appears that only the Config API and solrconfig.xml support loading custom > classes from the Blob Store. It seems to me like any directive for a > collection which references a class attribute should also support loading > this class from the Blob Store via a runtimeLib="true" attribute. > The obvious use case here is custom analyzers, but similarity is also a > candidate. > The documentation in this area (eg. "add-runtimelib") is pretty vague. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-10679) Solr CDCR cannot be configured to use Aliases for replication
Webster Homer created SOLR-10679: Summary: Solr CDCR cannot be configured to use Aliases for replication Key: SOLR-10679 URL: https://issues.apache.org/jira/browse/SOLR-10679 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: CDCR Affects Versions: 6.2 Reporter: Webster Homer My company uses Solr aliases to limit the configuration changes that we need to support. The CDCR configuration seems to accept an alias for either the source or target collections, and no errors show up in the log, but no data is replicated if the source or target is an alias and not an actual collection. I see that aliases are not even mentioned in the CDCR documentation. It seems to me this should either work or throw an error. It should be documented one way or another -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9940) Config API throws Index Not Mutable when creating a RequestHandler on a Classic Schema
[
https://issues.apache.org/jira/browse/SOLR-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Webster Homer updated SOLR-9940:
Priority: Minor (was: Major)
> Config API throws Index Not Mutable when creating a RequestHandler on a
> Classic Schema
> --
>
> Key: SOLR-9940
> URL: https://issues.apache.org/jira/browse/SOLR-9940
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: config-api
>Affects Versions: 6.2
> Environment: linux
>Reporter: Webster Homer
>Priority: Minor
>
> All of our schema are Classic, we do no use mutable schemas.
> We use the Config API to create /cdcr request handlers
> {code}
> {
> "add-requesthandler":{
> "name":"/cdcr",
> "class":"solr.CdcrRequestHandler",
> "replica": [{
> "zkHost":"stldeepx20.sial.com:2181/solr",
> "source":"sial-catalog-material",
> "target":"sial-catalog-material"
> }, {
> "zkHost":"stldeepx06.sial.com:2181/solr",
> "source":"sial-catalog-material",
> "target":"sial-catalog-material"
> }],
> "replicator": {
> "threadPoolSize":2,
> "schedule": 1000,
> "batchSize": 128
> },
> "updateLogSynchronizer" : {
> "schedule": 6
> }
> }
> }
> {code}
> The actual handler is generated by by an endpoint.
> The handler is created, and is functional, but the following errors are
> present in the log:
> {code}
> 2017-01-06 21:34:12.377 ERROR (SolrConfigHandler-refreshconf)
> [c:test-catalog-product s:shard1 r:core_node1 x:test-catalog-product_s
> hard1_replica1] o.a.s.s.IndexSchema This IndexSchema is not mutable.
> 2017-01-06 21:34:12.377 WARN (SolrConfigHandler-refreshconf)
> [c:test-catalog-product s:shard1 r:core_node1 x:test-catalog-product_s
> hard1_replica1] o.a.s.c.SolrCore
> org.apache.solr.common.SolrException: This IndexSchema is not mutable.
> at
> org.apache.solr.schema.IndexSchema.getSchemaUpdateLock(IndexSchema.java:1864)
> at
> org.apache.solr.schema.SchemaManager.getFreshManagedSchema(SchemaManager.java:434)
> at
> org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2571)
> at org.apache.solr.core.SolrCore$$Lambda$91/766533008.run(Unknown
> Source)
> at
> org.apache.solr.handler.SolrConfigHandler$Command.lambda$handleGET$0(SolrConfigHandler.java:216)
> at
> org.apache.solr.handler.SolrConfigHandler$Command$$Lambda$143/2029435585.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9940) Config API throws Index Not Mutable when creating a RequestHandler on a Classic Schema
Webster Homer created SOLR-9940:
---
Summary: Config API throws Index Not Mutable when creating a
RequestHandler on a Classic Schema
Key: SOLR-9940
URL: https://issues.apache.org/jira/browse/SOLR-9940
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: config-api
Affects Versions: 6.2
Environment: linux
Reporter: Webster Homer
All of our schema are Classic, we do no use mutable schemas.
We use the Config API to create /cdcr request handlers
{code}
{
"add-requesthandler":{
"name":"/cdcr",
"class":"solr.CdcrRequestHandler",
"replica": [{
"zkHost":"stldeepx20.sial.com:2181/solr",
"source":"sial-catalog-material",
"target":"sial-catalog-material"
}, {
"zkHost":"stldeepx06.sial.com:2181/solr",
"source":"sial-catalog-material",
"target":"sial-catalog-material"
}],
"replicator": {
"threadPoolSize":2,
"schedule": 1000,
"batchSize": 128
},
"updateLogSynchronizer" : {
"schedule": 6
}
}
}
{code}
The actual handler is generated by by an endpoint.
The handler is created, and is functional, but the following errors are present
in the log:
{code}
2017-01-06 21:34:12.377 ERROR (SolrConfigHandler-refreshconf)
[c:test-catalog-product s:shard1 r:core_node1 x:test-catalog-product_s
hard1_replica1] o.a.s.s.IndexSchema This IndexSchema is not mutable.
2017-01-06 21:34:12.377 WARN (SolrConfigHandler-refreshconf)
[c:test-catalog-product s:shard1 r:core_node1 x:test-catalog-product_s
hard1_replica1] o.a.s.c.SolrCore
org.apache.solr.common.SolrException: This IndexSchema is not mutable.
at
org.apache.solr.schema.IndexSchema.getSchemaUpdateLock(IndexSchema.java:1864)
at
org.apache.solr.schema.SchemaManager.getFreshManagedSchema(SchemaManager.java:434)
at
org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2571)
at org.apache.solr.core.SolrCore$$Lambda$91/766533008.run(Unknown
Source)
at
org.apache.solr.handler.SolrConfigHandler$Command.lambda$handleGET$0(SolrConfigHandler.java:216)
at
org.apache.solr.handler.SolrConfigHandler$Command$$Lambda$143/2029435585.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4164) Result Grouping fails if no hits
[
https://issues.apache.org/jira/browse/SOLR-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542906#comment-15542906
]
Webster Homer commented on SOLR-4164:
-
I am certain that in Solr 4.10 non-cloud specifying -1 did roll up all matching
documents. We certainly had more than 1 in the group.
I agree that at a minimum that solr should be consistent and either throw an
error or do something reasonable.
It would be nice to have a way to specify an unlimited roll up
> Result Grouping fails if no hits
>
>
> Key: SOLR-4164
> URL: https://issues.apache.org/jira/browse/SOLR-4164
> Project: Solr
> Issue Type: Bug
> Components: SearchComponents - other, SolrCloud
>Affects Versions: 4.0
>Reporter: Lance Norskog
>Assignee: Cao Manh Dat
>
> In SolrCloud, found a result grouping bug in the 4.0 release.
> A distributed result grouping request under SolrCloud got this result:
> {noformat}
> Dec 10, 2012 10:32:07 PM org.apache.solr.common.SolrException log
> SEVERE: null:java.lang.IllegalArgumentException: numHits must be > 0; please
> use TotalHitCountCollector if you just need the total hit count
> at
> org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:1120)
> at
> org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:1069)
> at
> org.apache.lucene.search.grouping.AbstractSecondPassGroupingCollector.(AbstractSecondPassGroupingCollector.java:75)
> at
> org.apache.lucene.search.grouping.term.TermSecondPassGroupingCollector.(TermSecondPassGroupingCollector.java:49)
> at
> org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.create(TopGroupsFieldCommand.java:128)
> at
> org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:132)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:339)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:206)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4164) Result Grouping fails if no hits
[
https://issues.apache.org/jira/browse/SOLR-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15178532#comment-15178532
]
Webster Homer commented on SOLR-4164:
-
Found that if you set group.limit to -1 it will give the same failure. We had
code that did this with the intent of getting all the documents in the roll up.
We limit it to 500 so setting group.limit=500 was a decent work around. Still
this worked fine with normal solr, only solrcloud had a problem
> Result Grouping fails if no hits
>
>
> Key: SOLR-4164
> URL: https://issues.apache.org/jira/browse/SOLR-4164
> Project: Solr
> Issue Type: Bug
> Components: SearchComponents - other, SolrCloud
>Affects Versions: 4.0
>Reporter: Lance Norskog
>
> In SolrCloud, found a result grouping bug in the 4.0 release.
> A distributed result grouping request under SolrCloud got this result:
> {noformat}
> Dec 10, 2012 10:32:07 PM org.apache.solr.common.SolrException log
> SEVERE: null:java.lang.IllegalArgumentException: numHits must be > 0; please
> use TotalHitCountCollector if you just need the total hit count
> at
> org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:1120)
> at
> org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:1069)
> at
> org.apache.lucene.search.grouping.AbstractSecondPassGroupingCollector.(AbstractSecondPassGroupingCollector.java:75)
> at
> org.apache.lucene.search.grouping.term.TermSecondPassGroupingCollector.(TermSecondPassGroupingCollector.java:49)
> at
> org.apache.solr.search.grouping.distributed.command.TopGroupsFieldCommand.create(TopGroupsFieldCommand.java:128)
> at
> org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:132)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:339)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:206)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
