bq: Is there a way to not write to disk continuously and only write the file...
Not if we're talking about the transaction log. The design is for the transaction log in particular to continuously get updates flushed to it, otherwise you could not replay the transaction log upon restart and have any hope of not losing data. And one other thing you want to be aware of: Having such long delays between commits will mean that in the event of ungraceful shut-down (pull the plug, kill -9 and the like) the _entire_ set if documents sent since the last time you did a commit will be replayed from the tlog before the replica accepts incoming requests. As for <2> I haven't a clue. Best, Erick On Tue, Oct 17, 2017 at 8:40 AM, Nawab Zada Asad Iqbal <khi...@gmail.com> wrote: > I take my yesterday's comment back. I assumed that the file being written > is a segment, however after letting solr run for the night. I see that the > segment is flushed at the expected size:1945MB (so that file which i > observed was still open for writing). > Now, I have two other questions:- > > 1. Is there a way to not write to disk continuously and only write the file > when segment is flushed? > > 2. With 6.5: i had ramBufferSizeMB=20G and limiting the threadCount to 12 > (since LUCENE-6659 > <https://issues.apache.org/jira/plugins/servlet/mobile#issue/LUCENE-6659>, > there is no configuration for indexing thread count, so I did a local > workaround to limit the number of threads in code); I had very good write > throughput. But with 7.0, I am getting comparable throughput only at > indexing threadcount > 50. What could be wrong ? > > > Thanks @Erick, I checked the commit settings, both soft and hard commits > are off. > > > > > On Tue, Oct 17, 2017 at 3:47 AM, Amrit Sarkar <sarkaramr...@gmail.com> > wrote: > >> > >> > In 7.0, i am finding that the file is written to disk very early on >> > and it is being updated every second or so. Had something changed in 7.0 >> > which is causing it? I tried something similar with solr 6.5 and i was >> > able to get almost a GB size files on disk. >> >> >> Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting >> 20GB segments on 6.5 or less? a GB? >> >> Amrit Sarkar >> Search Engineer >> Lucidworks, Inc. >> 415-589-9269 >> www.lucidworks.com >> Twitter http://twitter.com/lucidworks >> LinkedIn: https://www.linkedin.com/in/sarkaramrit2 >> >> On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal <khi...@gmail.com> >> wrote: >> >> > Hi, >> > >> > I have tuned (or tried to tune) my settings to only flush the segment >> > when it has reached its maximum size. At the moment,I am using my >> > application with only a couple of threads (i have limited to one thread >> for >> > analyzing this scenario) and my ramBufferSizeMB=20000 (i.e. ~20GB). With >> > this, I assumed that my file sizes on the disk will be at in the order of >> > GB; and no segments will be flushed until the segment's in memory size is >> > 2GB. In 7.0, i am finding that the file is written to disk very early on >> > and it is being updated every second or so. Had something changed in 7.0 >> > which is causing it? I tried something similar with solr 6.5 and i was >> > able to get almost a GB size files on disk. >> > >> > How can I control it to not write to disk until the segment has reached >> its >> > maximum permitted size (1945 MB?) ? My write traffic is 'new only' (i.e., >> > it doesn't delete any document) , however I also found following >> infostream >> > logs, which incorrectly say 'delete=true': >> > >> > Oct 16, 2017 10:18:29 PM INFO (qtp761960786-887) [ x:filesearch] >> > o.a.s.c.S.Request [filesearch] webapp=/solr path=/update >> > params={commit=false} status=0 QTime=21 >> > Oct 16, 2017 10:18:29 PM INFO (qtp761960786-889) [ x:filesearch] >> > o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges? >> > numDocsInRam=4434 deletes=true hasTickets:false >> pendingChangesInFullFlush: >> > false >> > Oct 16, 2017 10:18:29 PM INFO (qtp761960786-889) [ x:filesearch] >> > o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent: >> infoVersion >> > matches: false; DW changes: true; BD changes: false >> > Oct 16, 2017 10:18:29 PM INFO (qtp761960786-889) [ x:filesearch] >> > o.a.s.c.S.Request [filesearch] webapp=/solr path=/admin/luke >> > params={show=index&numTerms=0&wt=json} status=0 QTime=0 >> > >> > >> > >> > Thanks >> > Nawab >> > >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
