Brian, If you are still facing the issue after disabling buffer, kindly shut down all the nodes at source and then start them again, stale tlogs will start purging themselves.
Amrit Sarkar Search Engineer Lucidworks, Inc. 415-589-9269 www.lucidworks.com Twitter http://twitter.com/lucidworks LinkedIn: https://www.linkedin.com/in/sarkaramrit2 Medium: https://medium.com/@sarkaramrit2 On Wed, Jun 20, 2018 at 8:15 PM, Susheel Kumar <susheel2...@gmail.com> wrote: > Not in my knowledge. Please double check or wait for some time but after > DISABLEBUFFER on source, your logs should start rolling and its the exact > same issue I have faced with 6.6 which you resolve by DISABLEBUFFER. > > On Tue, Jun 19, 2018 at 1:39 PM, Brian Yee <b...@wayfair.com> wrote: > > > Does anyone have any additional possible causes for this issue? I checked > > the buffer status using "/cdcr?action=STATUS" and it says buffer disabled > > at both target and source. > > > > -----Original Message----- > > From: Erick Erickson [mailto:erickerick...@gmail.com] > > Sent: Tuesday, June 19, 2018 11:55 AM > > To: solr-user <solr-user@lucene.apache.org> > > Subject: Re: tlogs not deleting > > > > bq. Do you recommend disabling the buffer on the source SolrCloud as > well? > > > > Disable them all on both source and target IMO. > > > > On Tue, Jun 19, 2018 at 8:50 AM, Brian Yee <b...@wayfair.com> wrote: > > > Thank you Erick. I am running Solr 6.6. From the documentation: > > > "Replicas do not need to buffer updates, and it is recommended to > > disable buffer on the target SolrCloud." > > > > > > Do you recommend disabling the buffer on the source SolrCloud as well? > > It looks like I already have the buffer disabled at target locations but > > not the source location. Would it even make sense at the source location? > > > > > > This is what I have at the target locations: > > > <requestHandler name="/cdcr" class="solr.CdcrRequestHandler"> > > > <shardHandlerFactory class="HttpShardHandlerFactory"> > > > <int name="maxConnectionsPerHost">100</int> > > > </shardHandlerFactory> > > > <lst name="buffer"> > > > <str name="defaultState">disabled</str> > > > </lst> > > > </requestHandler> > > > > > > > > > -----Original Message----- > > > From: Erick Erickson [mailto:erickerick...@gmail.com] > > > Sent: Tuesday, June 19, 2018 11:00 AM > > > To: solr-user <solr-user@lucene.apache.org> > > > Subject: Re: tlogs not deleting > > > > > > Take a look at the CDCR section of your reference guide, be sure you > get > > the version which you can download from here: > > > https://archive.apache.org/dist/lucene/solr/ref-guide/ > > > > > > There's the CDCR API call you can use for in-flight disabling, and > > depending on the version of Solr you can set it in solrconfig. > > > > > > Basically, buffering was there in the original CDCR to allow a larger > > maintenance window, you could enable buffering and all updates were saved > > until you disabled it, during which period you could do whatever you > needed > > with your target cluster and not lose any updates. > > > > > > Later versions can do the full sync of the index and buffering is being > > removed. > > > > > > Best, > > > Erick > > > > > > On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee <b...@wayfair.com> wrote: > > >> Thanks for the suggestion. Can you please elaborate a little bit about > > what DISABLEBUFFER does? The documentation is not very detailed. Is this > > something that needs to be done manually whenever this problem happens or > > is it something that we can do to fix it so it won't happen again? > > >> > > >> -----Original Message----- > > >> From: Susheel Kumar [mailto:susheel2...@gmail.com] > > >> Sent: Monday, June 18, 2018 9:12 PM > > >> To: solr-user@lucene.apache.org > > >> Subject: Re: tlogs not deleting > > >> > > >> You may have to DISABLEBUFFER in source to get rid of tlogs. > > >> > > >> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee <b...@wayfair.com> wrote: > > >> > > >>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I > > >>> understand, after a hard commit, solr is supposed to delete old > > >>> tlogs depending on the numRecordsToKeep and maxNumLogsToKeep values > > >>> in the autocommit settings in solrconfig.xml. I am occasionally > > >>> seeing solr fail to do this and the tlogs just build up over time > > >>> and eventually we run out of disk space on the VM and this causes > > problems for us. > > >>> This does not happen all the time, only sometimes. I currently have > > >>> a tlog directory that has 123G worth of tlogs. The last hard commit > > >>> on this node was 10 minutes ago but these tlogs date back to 3 days > > ago. > > >>> > > >>> We have sometimes found that restarting solr on the node will get it > > >>> to clean up the old tlogs, but we really want to find the root cause > > >>> and fix it if possible so we don't keep getting disk space alerts > > >>> and have to adhoc restart nodes. Has anyone seen an issue like this > > before? > > >>> > > >>> My update handler settings look like this: > > >>> <updateHandler class="solr.DirectUpdateHandler2"> > > >>> > > >>> <updateLog class="solr.CdcrUpdateLog"> > > >>> > > >>> <str name="dir">${solr.ulog.dir:}</str> > > >>> <int name="numVersionBuckets">${solr.ulog.numVersionBuckets: > > >>> 65536}</int> > > >>> </updateLog> > > >>> <autoCommit> > > >>> <maxTime>600000</maxTime> > > >>> <maxDocs>250000</maxDocs> > > >>> <openSearcher>false</openSearcher> > > >>> </autoCommit> > > >>> <autoSoftCommit> > > >>> <maxTime>120000</maxTime> > > >>> </autoSoftCommit> > > >>> > > >>> <shardHandlerFactory class="HttpShardHandlerFactory"> > > >>> <int name="maxConnectionsPerHost">100</int> > > >>> </shardHandlerFactory> > > >>> > > >>> </updateHandler> > > >>> > > >
