Re: CDCR - how to deal with the transaction log files
Thanks for the suggestion - tried that today and still no luck. Time to write a script to naively / blindly delete old logs and run that in cron. *sigh* -- View this message in context: http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4346138.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: CDCR - how to deal with the transaction log files
glad to hear you found your solution! I have been combing over this post and
others on this discussion board many times and have tried so many tweaks to
configuration, order of steps, etc, all with absolutely no success in
getting the Source cluster tlogs to delete. So incredibly frustrating. If
anyone has other pearls of wisdom I'd love some advice. Quick hits on what
I've tried:
- solrconfig exactly like Sean's (target and source respectively) expect no
autoSoftCommit
- I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
target) explicitly before starting since the config setting of
defaultState=disabled doesn't seem to work
- when I create the collection on source first, I get the warning "The log
reader for target collection {collection name} is not initialised". When I
reverse the order (create the collection on target first), no such warning
- tlogs replicate as expected, hard commits on both target and source cause
tlogs to rollover, etc - all of that works as expected
- action=QUEUES on source reflects the queueSize accurately. Also *always*
shows updateLogSynchronizer state as "stopped"
- action=LASTPROCESSEDVERSION on both source and target always seems correct
(I don't see the -1 that Sean mentioned).
- I'm creating new collections every time and running full data imports that
take 5-10 minutes. Again, all data replication, log rollover, and autocommit
activity seems to work as expected, and logs on target are deleted. It's
just those pesky source tlogs I can't get to delete.
--
View this message in context:
http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4345715.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tlogs not being deleted/truncated
another interesting clue in my case (different from what WebsterHomer is seeing): the response from /cdcr?action=QUEUES reflects what I would expect to see in the tlog directory but it's not accurate. By that I mean tlogTotalSize shows 1500271 (bytes) and tlogTotalCount shows 2. This changes as more updates come in and autoCommit runs - sometimes tlogTotalCount is 1 instead of 2, and the tlogTotalSize changes but stays in that low range. But on the filesystem, all the tlogs are still there. Perhaps the ignored exception noted above is in fact a problem? -- View this message in context: http://lucene.472066.n3.nabble.com/Tlogs-not-being-deleted-truncated-tp4341958p4345477.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Tlogs not being deleted/truncated
Have you found a solution to this? I'm seeing the same behavior (running solr v6.5.0): - source data center replicates to target just fine - autocommit on Source with openSearcher=false causes tlogs to rollover at a reasonable size - a post-import explicit commit (with the default openSearcher=true) causes docs to be visible on Source - autocommit on Target with openSearcher=true (and a higher maxTime / maxDocs) causes tlogs to roll eventually and docs to become visible without any need for a final explicit commit - tlogs on Target clean up (get deleted) just fine, after each hard commit - *tlogs on Source are never deleted and I can't figure out why* I've tried this with the cdcr buffer both enabled and disabled, and tried it with DataImportHandler imports as well as SolrJ posts of updates. I can't find any combination of settings that causes tlogs on the Source cluster to be cleaned up. Any ideas on how to manage this other than implement a manual log deletion routine on the filesystem? -- View this message in context: http://lucene.472066.n3.nabble.com/Tlogs-not-being-deleted-truncated-tp4341958p4345473.html Sent from the Solr - User mailing list archive at Nabble.com.
