Re: commit in solr4 takes a longer time
Hi sandeep, I made the changes u mentioned and tested again for the same set of docs but unfortunately the commit time increased. -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060622.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi Gopal, I added the opensearcher parameter as mentioned by you but on checking logs I found that apensearcher was still true on commit. it is only when I removed the autosoftcommit parameter the opensearcher parameter worked and provided faster updates as well. however I require soft commit in my application. Any suggestions. -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060623.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
That's not ideal. Can you post solrconfig.xml? On 3 May 2013 07:41, vicky desai vicky.de...@germinait.com wrote: Hi sandeep, I made the changes u mentioned and tested again for the same set of docs but unfortunately the commit time increased. -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060622.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler /
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060628.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi All, setting opensearcher flag to true solution worked and it give me visible improvement in commit time. One thing to make note of is that while using solrj client we have to call server.commit(false,false) which i was doing incorrectly and hence was not able to see the improvement earliear. Thanks everyone -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060688.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi,
After using the following config
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5000/maxDocs
openSearcherfalse/openSearcher
/autoCommit
/updateHandler
When a commit operation is fired I am getting the following logs
INFO: start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
even though openSearcher is false , waitSearcher is true . Can that be set
to false too? Will that give a performance improvement and what is the
config for that
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060706.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Since you have define commit option as Auto Commit for hard and soft commit then you don't have to explicitly call commit from SolrJ client. And openSearcher=false for hard commit will make hard commit faster since it is only makes sure that recent changes are flushed to disk (for durability) and not opening any searcher. can you post you log when soft commit and hard commit happens? You can read about waitFlush=false and waitSearcher=false which are default to true, see below from java doc JavaDoc: *waitFlush* block until index changes are flushed to disk *waitSearcher* block until a new searcher is opened and registered as the main query searcher, making the changes visible*T* On Fri, May 3, 2013 at 7:19 AM, vicky desai vicky.de...@germinait.comwrote: Hi All, setting opensearcher flag to true solution worked and it give me visible improvement in commit time. One thing to make note of is that while using solrj client we have to call server.commit(false,false) which i was doing incorrectly and hence was not able to see the improvement earliear. Thanks everyone -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060688.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi,
When a auto commit operation is fired I am getting the following logs
INFO: start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
setting the openSearcher to false definetly gave me a lot of performance
improvement but was wondering if waitSearcher can also be set to false and
will that give me a performance raise too.
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060715.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
On 5/3/2013 9:28 AM, vicky desai wrote:
Hi,
When a auto commit operation is fired I am getting the following logs
INFO: start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
setting the openSearcher to false definetly gave me a lot of performance
improvement but was wondering if waitSearcher can also be set to false and
will that give me a performance raise too.
The openSearcher parameter changes what actually happens when you do a
hard commit, so using it can change your performance.
The wait parameters are for client software that does commits. The
idea is that if you don't want your client to wait for the commit to
finish, you use these options so that the commit API call will return
quickly and the server will finish the commit in the background. It
doesn't change what the commit does, it just allows the client to start
doing other things.
With auto commits, the client and the server are both Solr, and
everything is multi-threaded. The wait parameters have no meaning,
because there's no user software that has to wait. There would be no
performance gain from turning them off.
Side note: The waitFlush parameter was completely removed in Solr 4.0.
Thanks,
Shawn
commit in solr4 takes a longer time
Hi all, I have recently migrated from solr 3.6 to solr 4.0. The documents in my core are getting constantly updated and so I fire a code commit after every 10 thousand docs . However moving from 3.6 to 4.0 I have noticed that for the same core size it takes about twice the time to commit in solr4.0 compared to solr 3.6. Is there any workaround by which I can reduce this time. Any help would be highly appreciated -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Can you explain more about your document size, shard and replica sizes, and auto/soft commit time parameters? 2013/5/2 vicky desai vicky.de...@germinait.com Hi all, I have recently migrated from solr 3.6 to solr 4.0. The documents in my core are getting constantly updated and so I fire a code commit after every 10 thousand docs . However moving from 3.6 to 4.0 I have noticed that for the same core size it takes about twice the time to commit in solr4.0 compared to solr 3.6. Is there any workaround by which I can reduce this time. Any help would be highly appreciated -- View this message in context: http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler /
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
First, I would upgrade to 4.2.1 and remember to change luceneMatchVersion to
LUCENE_42.
There were a LOT of fixes between 4.0 and 4.2.1.
wunder
On May 2, 2013, at 12:16 AM, vicky desai wrote:
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler /
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
Walter Underwood
wun...@wunderwood.org
Re: commit in solr4 takes a longer time
you might want to added openSearcher=false for hard commit, so hard commit
also act like soft commit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
On Thu, May 2, 2013 at 12:16 AM, vicky desai vicky.de...@germinait.comwrote:
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler /
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication
class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
What happens exactly when you don't open searcher at commit?
2013/5/2 Gopal Patwa gopalpa...@gmail.com
you might want to added openSearcher=false for hard commit, so hard commit
also act like soft commit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
On Thu, May 2, 2013 at 12:16 AM, vicky desai vicky.de...@germinait.com
wrote:
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard
class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler
/
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication
class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
If you don't re-open the searcher, you will not see new changes. So,
if you only have hard commit, you never see those changes (until
restart). But if you also have soft commit enabled, that will re-open
your searcher for you.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Thu, May 2, 2013 at 11:21 AM, Furkan KAMACI furkankam...@gmail.com wrote:
What happens exactly when you don't open searcher at commit?
2013/5/2 Gopal Patwa gopalpa...@gmail.com
you might want to added openSearcher=false for hard commit, so hard commit
also act like soft commit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
On Thu, May 2, 2013 at 12:16 AM, vicky desai vicky.de...@germinait.com
wrote:
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard
class=solr.StandardRequestHandler
default=true /
requestHandler name=/update class=solr.UpdateRequestHandler
/
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication
class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit in solr4 takes a longer time
Hi Vicky,
I faced this issue as well and after some playing around I found the
autowarm count in cache sizes to be a problem.
I changed that from a fixed count (3072) to percentage (10%) and all commit
times were stable then onwards.
filterCache class=solr.FastLRUCache size=8192 initialSize=3072
autowarmCount=10% /
queryResultCache class=solr.LRUCache size=16384 initialSize=3072
autowarmCount=10% /
documentCache class=solr.LRUCache size=8192 initialSize=4096
autowarmCount=10% /
HTH,
Sandeep
On 2 May 2013 16:31, Alexandre Rafalovitch arafa...@gmail.com wrote:
If you don't re-open the searcher, you will not see new changes. So,
if you only have hard commit, you never see those changes (until
restart). But if you also have soft commit enabled, that will re-open
your searcher for you.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Thu, May 2, 2013 at 11:21 AM, Furkan KAMACI furkankam...@gmail.com
wrote:
What happens exactly when you don't open searcher at commit?
2013/5/2 Gopal Patwa gopalpa...@gmail.com
you might want to added openSearcher=false for hard commit, so hard
commit
also act like soft commit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
On Thu, May 2, 2013 at 12:16 AM, vicky desai vicky.de...@germinait.com
wrote:
Hi,
I am using 1 shard and two replicas. Document size is around 6 lakhs
My solrconfig.xml is as follows
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
indexConfig
maxFieldLength2147483647/maxFieldLength
lockTypesimple/lockType
unlockOnStartuptrue/unlockOnStartup
/indexConfig
updateHandler class=solr.DirectUpdateHandler2
autoSoftCommit
maxDocs500/maxDocs
maxTime1000/maxTime
/autoSoftCommit
autoCommit
maxDocs5/maxDocs
maxTime30/maxTime
/autoCommit
/updateHandler
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=204800 /
/requestDispatcher
requestHandler name=standard
class=solr.StandardRequestHandler
default=true /
requestHandler name=/update
class=solr.UpdateRequestHandler
/
requestHandler name=/admin/
class=org.apache.solr.handler.admin.AdminHandlers /
requestHandler name=/replication
class=solr.ReplicationHandler /
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory} /
enableLazyFieldLoadingtrue/enableLazyFieldLoading
admin
defaultQuery*:*/defaultQuery
/admin
/config
--
View this message in context:
http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
Sent from the Solr - User mailing list archive at Nabble.com.
