Hi Thanks a lot for your answer, Erick!
I changed the value of the autoSoftCommit property and it had the expected effect. It can be noted that this is per Core, so I get four getReader calls when my Solr contains four cores per autoSoftCommit interval.
Is it correct that a segment file is ready for merging after a commit has been done (e.g. using the autoCommit property), so I will see merges of 100 and up documents (and the index writer continues writing into a new segment file)?
It looks like the segments are being merged into 6 MB files and when enough into 60MB files and these again into 3,5GB files.
Best regards Trym Den 19-09-2012 14:49, Erick Erickson skrev:
I _think_ the getReader calls are being triggered by the autoSoftCommit being at one second. If so, this is probably OK. But bumping that up would nail whether that's the case... About RamBufferSizeMB. This has nothing to do with the size of the segments! It's just how much memory is consumed before the RAMBuffer is flushed to the _currently open_ segment. So until a hard commit happens, the currently open segment will continue to grow as successive RAMBuffers are flushed. bq: I expected that my Lucene index segment files would be a bit bigger than 1KB Is this a typo? The 512 is specifying MB...... Best Erick On Wed, Sep 19, 2012 at 6:01 AM, "Trym R. Møller" <t...@sigmat.dk> wrote:Hi Using SolrCloud I have added the following to solrconfig.xml (actually the node in zookeeper) <ramBufferSizeMB>512</ramBufferSizeMB> After that I expected that my Lucene index segment files would be a bit bigger than 1KB as I'm indexing very small documents Enabling the infoStream I see a lot of "flush at getReader" (one segment of the infoStream file pasted below) 1. Where can I look for why documents are flushed so frequently? 2. Does it have anything to do with "getReader" and can I do anything so Solr doesn't need to get a new reader so often? Any comments are most welcome. Best regards Trym Furthermore I have specified <autoCommit> <maxTime>180000</maxTime> </autoCommit> <autoSoftCommit> <maxTime>1000</maxTime> </autoSoftCommit> IW 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: flush at getReader DW 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: pool-12-thread-1 startFullFlush DW 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: anyChanges? numDocsInRam=7 deletes=false hasTickets:false pendingChangesInFullFlush: false DWFC 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_kc, aborting=false, numDocsInRAM=7, deleteQueue=DWDQ: [ generation: 1 ]] DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: flush postings as segment _kc numDocs=7 DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: new segment has 0 deleted docs DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: new segment has no vectors; norms; no docValues; prox; freqs DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: flushedFiles=[_kc_Lucene40_0.frq, _kc.fnm, _kc_Lucene40_0.tim, _kc_nrm.cfs, _kc.fdx, _kc.fdt, _kc_Lucene40_0.prx, _kc_nrm.cfe, _kc_Lucene40_0.tip] DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: flushed codec=Lucene40 DWPT 0 [Wed Sep 19 11:07:45 CEST 2012; pool-12-thread-1]: flushed: segment=_kc ramUsed=0,095 MB newFlushedSize(includes docstores)=0,003 MB docs/MB=2.283,058
