Re: Soft commit and flush

2013-10-07 Thread Erick Erickson
bq: If so, using soft commit without calling hard commit could cause OOM no. Aside from anything you have configured for auto(hard) commit, the ramBufferSizeMB in solrconfig.xml will flush the in-memory structures out to the segments when the size reaches this limit. It won't _close_ the current

Re: Soft commit and flush

2013-10-07 Thread Guido Medina
Out of Memory Exception is well known as OOM. Guido. On 07/10/13 14:11, adfel70 wrote: Sorry, by "OOE" I meant Out of memory exception... -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html Sent from the Solr - User mailing list a

Re: Soft commit and flush

2013-10-07 Thread adfel70
Sorry, by "OOE" I meant Out of memory exception... -- View this message in context: http://lucene.472066.n3.nabble.com/Soft-commit-and-flush-tp4091726p4093902.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Soft commit and flush

2013-10-07 Thread Erick Erickson
bq: Does the NRTCachingDirectoryFactory relevant for both types of commit, or just for hard commit Don't know the code deeply, but NRT==Near Real Time == Soft commit I'd guess. bq: If soft commit does not flush... soft commit flushes the transaction log. On restart if the content of the tlog isn

Re: Soft commit and flush

2013-10-06 Thread adfel70
I understand the bottom line that soft commits are about visibility, hard commits are about durability. I am just trying to gain better understanding what happens under the hood... 2 more related questions you made me think of: 1. Does the NRTCachingDirectoryFactory relevant for both types of commi

Re: Soft commit and flush

2013-09-25 Thread Erick Erickson
Why do you care? Curiosity or are you trying to find a behavior you can count on? because "soft commits are about visibility, hard commits are about durability". Meaning you can't count on a soft commit writing anything to disk at all. I suspect in your tests the soft commit had nothing to do with

Re: Soft commit and flush

2013-09-24 Thread Shawn Heisey
On 9/24/2013 5:51 AM, adfel70 wrote: My conclusion is that soft commit always flushes the data, but because of the implementation of NRTCachingDirectoryFactory, the data will be written to the disk when its getting too big. The NRTCachingDirectoryFactory (which creates NRTCachingDirectory inst

Re: Soft commit and flush

2013-09-24 Thread Otis Gospodnetic
Hi, I believe data is not fsynched to disk until a hard commit (and even then disks can lie to you and tell you data is safe even though it's still in disk cache waiting to really be written to the medium) , which is why you can lose it between hard commits. Soft commits just make newly added doc