So I'm continuing to look into this and not making much headway, but I have
additional questions now as well.
I restarted the nodes in the source data center to see if it would have any
impact. It appeared to initiate another bootstrap with the target. The lag and
queueSize were brought back down to zero.
Over the next two hours the queueSize has grown back to 106,122 (as reported by
solr/mycollection/cdcr?action=QUEUES). When I actually look at what we sent to
Solr though, I only deleted or added a total of 3,805 documents. Could this be
part of the problem? Should queueSize be representative of the total number of
document updates, or are there other updates under the hood that I wouldn't see
that would still need to be tracked by Solr.
Also, if there are any other suggestions on my original issue which is that the
CDCR cannot keep up despite the relatively low number of updates (3805 over two
hours).
Thanks.
> On Mar 7, 2018, at 6:19 PM, Tom Peters <tpet...@synacor.com> wrote:
>
> I'm having issues with the target collection staying up-to-date with indexing
> from the source collection using CDCR.
>
> This is what I'm getting back in terms of OPS:
>
> curl -s 'solr2-a:8080/solr/mycollection/cdcr?action=OPS' | jq .
> {
> "responseHeader": {
> "status": 0,
> "QTime": 0
> },
> "operationsPerSecond": [
> "zook01,zook02,zook03/solr",
> [
> "mycollection",
> [
> "all",
> 49.10140553500938,
> "adds",
> 10.27612635309587,
> "deletes",
> 38.82527896994054
> ]
> ]
> ]
> }
>
> The source and target collections are in separate data centers.
>
> Doing a network test between the leader node in the source data center and
> the ZooKeeper nodes in the target data center
> show decent enough network performance: ~181 Mbit/s
>
> I've tried playing around with the "batchSize" value (128, 512, 728, 1000,
> 2000, 2500) and they've haven't made much of a difference.
>
> Any suggestions on potential settings to tune to improve the performance?
>
> Thanks
>
> --
>
> Here's some relevant log lines from the source data center's leader:
>
> 2018-03-07 23:16:11.984 INFO
> (cdcr-replicator-207-thread-3-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
> 2018-03-07 23:16:23.062 INFO
> (cdcr-replicator-207-thread-4-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 510 updates to target mycollection
> 2018-03-07 23:16:32.063 INFO
> (cdcr-replicator-207-thread-5-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
> 2018-03-07 23:16:36.209 INFO
> (cdcr-replicator-207-thread-1-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection
> 2018-03-07 23:16:42.091 INFO
> (cdcr-replicator-207-thread-2-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection
> 2018-03-07 23:16:46.790 INFO
> (cdcr-replicator-207-thread-3-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
> 2018-03-07 23:16:50.004 INFO
> (cdcr-replicator-207-thread-4-processing-n:solr2-a:8080_solr
> x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
> [c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
> o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection
>
>
> And what the log looks like in the target:
>
> 2018-03-07 23:18:46.475 INFO (qtp1595212853-26) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067896487950&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.500 INFO (qtp1595212853-25) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067896487951&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.525 INFO (qtp1595212853-24) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536512&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.550 INFO (qtp1595212853-3793) [c:mycollection
> s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536513&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.575 INFO (qtp1595212853-30) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536514&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.600 INFO (qtp1595212853-26) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536515&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.625 INFO (qtp1595212853-25) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536516&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.651 INFO (qtp1595212853-24) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536517&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.676 INFO (qtp1595212853-3793) [c:mycollection
> s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536518&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
> 2018-03-07 23:18:46.701 INFO (qtp1595212853-30) [c:mycollection s:shard1
> r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
> [mycollection_shard1_replica_n1] webapp=/solr path=/update
> params={_stateVer_=mycollection:30&_version_=-1594317067897536519&cdcr.update=&wt=javabin&version=2}
> status=0 QTime=0
>
>
>
> This message and any attachment may contain information that is confidential
> and/or proprietary. Any use, disclosure, copying, storing, or distribution of
> this e-mail or any attached file by anyone other than the intended recipient
> is strictly prohibited. If you have received this message in error, please
> notify the sender by reply email and delete the message and any attachments.
> Thank you.
This message and any attachment may contain information that is confidential
and/or proprietary. Any use, disclosure, copying, storing, or distribution of
this e-mail or any attached file by anyone other than the intended recipient is
strictly prohibited. If you have received this message in error, please notify
the sender by reply email and delete the message and any attachments. Thank you.
