Instead of CDCR you may simply duplicate the pipeline across both data centers. Then there is no need at each step of the pipeline to replicate (storage to storage, index to index etc.). Instead both pipelines run in different data centers in parallel.
> Am 24.06.2020 um 15:46 schrieb Oakley, Craig (NIH/NLM/NCBI) [C] > <craig.oak...@nih.gov.invalid>: > > In attempting to stress-test CDCR (running Solr 7.4), I am running into a > couple of issues. > > One is that the tlog files keep accumulating for some nodes in the CDCR > system, particularly for the non-Leader nodes in the Source SolrCloud. No > quantity of hard commits seem to cause any of these tlog files to be > released. This can become a problem upon reboot if there are hundreds of > thousands of tlog files, and Solr fails to start (complaining that there are > too many open files). > > The tlogs had been accumulating on all the nodes of the CDCR set of > SolrClouds until I added these two lines to the solrconfig.xml file (for > testing purposes, using numbers much lower than in the examples): > <int name="numRecordsToKeep">5</int> > <int name="maxNumLogsToKeep">2</int> > Since then, it is mostly the non-Leader nodes of the Source SolrCloud which > accumulates tlog files (the Target SolrCloud does seem to have a tendency to > clean up the tlog files, as does the Leader of the Source SolrCloud). If I > use ADDREPLICAPROP and REBALANCELEADERS to change which node is the Leader, > and if I then start adding more data, the tlogs on the new Leader sometimes > will go away, but then the old Leader begins accumulating tlog files. I am > dubious whether frequent reassignment of Leadership would be a practical > solution. > > I also have several times attempted to simulate a production environment by > running several loops simultaneously, each of which inserts multiple records > on each iteration of the loop. Several times, I end up with a dozen records > on (both replicas of) the Source which never make it to (either replica of) > the Target. The Target has thousands of records which were inserted before > the missing records, and thousands of records which were inserted after the > missing records (and all these records, the replicated and the missing, were > inserted by curl commands which only differed in sequential numbers > incorporated into the values being inserted). > > I also have a question regarding SOLR-13141: the 11/Feb/19 comment says that > the fix for Solr 7.3 had a problem; and the header says "Affects Version/s: > 7.5, 7.6": does that indicate that Solr 7.4 is not affected? > > Are there any suggestions? > > Thanks
