RE: Do commits block updates in SOLR 1.4?
So you are saying we definitely do not need to pause ADD activity on other threads while we send the COMMIT? And the same goes with AUTOCOMMIT right? We are using SOLR 1.4 now. We were on 1.3 previously. We pretty much just assumed pausing ADDs during COMMITs was required by SOLR when we designed our indexing system, mostly due to our experience with an older and different search engine. -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Thursday, September 02, 2010 6:10 PM To: solr-user@lucene.apache.org Subject: Re: Do commits block updates in SOLR 1.4? Yes, indexing synchronized during commits. You can call commit all you want, and index docs, and commit will finish and then indexing will restart. Previous Solr release did this also; how far back is your existing Solr? On Thu, Sep 2, 2010 at 1:11 PM, Robert Petersen rober...@buy.com wrote: Hello sorry to bother but does anyone know the answer to this? This is the closest thing I can find on the subject: http://lucene.472066.n3.nabble.com/Autocommit-blocking-adds-AutoCommit-S peedup-td498465.html -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Wednesday, September 01, 2010 11:35 AM To: solr-user@lucene.apache.org Subject: Do commits block updates in SOLR 1.4? I can't seem to find a definitive answer. I have ten threads doing my indexing and I block all the threads when one is ready to do a commit so no adds are done until the commit finishes. Is this still required in SOLR 1.4 or could I take it out? I tried testing this on a separate small index where I set autocommit in solrconfig and seem to have no issues just continuously adding documents from multiple threads to it despite its commit activity. I'd like to do the same in my big main index, is it safe? Also, is there any difference in behavior between autocommits and explicit commits in this regard? -- Lance Norskog goks...@gmail.com
Re: Do commits block updates in SOLR 1.4?
Solr handles all of this concurrency for you - it's actually even a little too aggressive about that these days, as Lucene has changed a lot - but yes - you can add while committing and commit while adding - Solr will block itself as needed. - Mark On 9/3/10 1:27 PM, Robert Petersen wrote: So you are saying we definitely do not need to pause ADD activity on other threads while we send the COMMIT? And the same goes with AUTOCOMMIT right? We are using SOLR 1.4 now. We were on 1.3 previously. We pretty much just assumed pausing ADDs during COMMITs was required by SOLR when we designed our indexing system, mostly due to our experience with an older and different search engine. -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Thursday, September 02, 2010 6:10 PM To: solr-user@lucene.apache.org Subject: Re: Do commits block updates in SOLR 1.4? Yes, indexing synchronized during commits. You can call commit all you want, and index docs, and commit will finish and then indexing will restart. Previous Solr release did this also; how far back is your existing Solr? On Thu, Sep 2, 2010 at 1:11 PM, Robert Petersen rober...@buy.com wrote: Hello sorry to bother but does anyone know the answer to this? This is the closest thing I can find on the subject: http://lucene.472066.n3.nabble.com/Autocommit-blocking-adds-AutoCommit-S peedup-td498465.html -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Wednesday, September 01, 2010 11:35 AM To: solr-user@lucene.apache.org Subject: Do commits block updates in SOLR 1.4? I can't seem to find a definitive answer. I have ten threads doing my indexing and I block all the threads when one is ready to do a commit so no adds are done until the commit finishes. Is this still required in SOLR 1.4 or could I take it out? I tried testing this on a separate small index where I set autocommit in solrconfig and seem to have no issues just continuously adding documents from multiple threads to it despite its commit activity. I'd like to do the same in my big main index, is it safe? Also, is there any difference in behavior between autocommits and explicit commits in this regard?
RE: Do commits block updates in SOLR 1.4?
Thanks guys! I will be quite happy to remove the unnecessary complexity from our code. -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Friday, September 03, 2010 10:28 AM To: solr-user@lucene.apache.org Subject: Re: Do commits block updates in SOLR 1.4? Solr handles all of this concurrency for you - it's actually even a little too aggressive about that these days, as Lucene has changed a lot - but yes - you can add while committing and commit while adding - Solr will block itself as needed. - Mark On 9/3/10 1:27 PM, Robert Petersen wrote: So you are saying we definitely do not need to pause ADD activity on other threads while we send the COMMIT? And the same goes with AUTOCOMMIT right? We are using SOLR 1.4 now. We were on 1.3 previously. We pretty much just assumed pausing ADDs during COMMITs was required by SOLR when we designed our indexing system, mostly due to our experience with an older and different search engine. -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Thursday, September 02, 2010 6:10 PM To: solr-user@lucene.apache.org Subject: Re: Do commits block updates in SOLR 1.4? Yes, indexing synchronized during commits. You can call commit all you want, and index docs, and commit will finish and then indexing will restart. Previous Solr release did this also; how far back is your existing Solr? On Thu, Sep 2, 2010 at 1:11 PM, Robert Petersen rober...@buy.com wrote: Hello sorry to bother but does anyone know the answer to this? This is the closest thing I can find on the subject: http://lucene.472066.n3.nabble.com/Autocommit-blocking-adds-AutoCommit-S peedup-td498465.html -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Wednesday, September 01, 2010 11:35 AM To: solr-user@lucene.apache.org Subject: Do commits block updates in SOLR 1.4? I can't seem to find a definitive answer. I have ten threads doing my indexing and I block all the threads when one is ready to do a commit so no adds are done until the commit finishes. Is this still required in SOLR 1.4 or could I take it out? I tried testing this on a separate small index where I set autocommit in solrconfig and seem to have no issues just continuously adding documents from multiple threads to it despite its commit activity. I'd like to do the same in my big main index, is it safe? Also, is there any difference in behavior between autocommits and explicit commits in this regard?
RE: Do commits block updates in SOLR 1.4?
Hello sorry to bother but does anyone know the answer to this? This is the closest thing I can find on the subject: http://lucene.472066.n3.nabble.com/Autocommit-blocking-adds-AutoCommit-S peedup-td498465.html -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Wednesday, September 01, 2010 11:35 AM To: solr-user@lucene.apache.org Subject: Do commits block updates in SOLR 1.4? I can't seem to find a definitive answer. I have ten threads doing my indexing and I block all the threads when one is ready to do a commit so no adds are done until the commit finishes. Is this still required in SOLR 1.4 or could I take it out? I tried testing this on a separate small index where I set autocommit in solrconfig and seem to have no issues just continuously adding documents from multiple threads to it despite its commit activity. I'd like to do the same in my big main index, is it safe? Also, is there any difference in behavior between autocommits and explicit commits in this regard?
Re: Do commits block updates in SOLR 1.4?
Yes, indexing synchronized during commits. You can call commit all you want, and index docs, and commit will finish and then indexing will restart. Previous Solr release did this also; how far back is your existing Solr? On Thu, Sep 2, 2010 at 1:11 PM, Robert Petersen rober...@buy.com wrote: Hello sorry to bother but does anyone know the answer to this? This is the closest thing I can find on the subject: http://lucene.472066.n3.nabble.com/Autocommit-blocking-adds-AutoCommit-S peedup-td498465.html -Original Message- From: Robert Petersen [mailto:rober...@buy.com] Sent: Wednesday, September 01, 2010 11:35 AM To: solr-user@lucene.apache.org Subject: Do commits block updates in SOLR 1.4? I can't seem to find a definitive answer. I have ten threads doing my indexing and I block all the threads when one is ready to do a commit so no adds are done until the commit finishes. Is this still required in SOLR 1.4 or could I take it out? I tried testing this on a separate small index where I set autocommit in solrconfig and seem to have no issues just continuously adding documents from multiple threads to it despite its commit activity. I'd like to do the same in my big main index, is it safe? Also, is there any difference in behavior between autocommits and explicit commits in this regard? -- Lance Norskog goks...@gmail.com
