Re: Commit is taking very long time
: My application is in prod and quite frequently�getting NullPointerException. ... : java.lang.NullPointerException : at com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.indexData(AuctionCollectionServiceImpl.java:251) : at com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.process(AuctionCollectionServiceImpl.java:135) : at com.fm.search.job.SearchIndexingJob.executeInternal(SearchIndexingJob.java:68) : at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86) : at org.quartz.core.JobRunShell.run(JobRunShell.java:202) : at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529) that stack trace doesn't suggest anything remotely related to Solr. none of those classes are in teh Solr code base -- without having any idea what the code on line 251 of your AuctionCollectionServiceImpl class looks like, no one could even begin to speculate what is causing the NPE. Even if we know what line 251 looks like, uinderstanding why some refrence on that line is null would probably require knowing a whole lot more about your application. -Hoss
Re: Commit is taking very long time
Hi,
Sorry in delaying to mail!
My application is in prod and quite frequently getting NullPointerException.
Initially I thought this is happening because of memory issue, so I reduced
mergeFactor to 5 and reduced number of document per commit to 2000. After these
changes for some time it stopped getting NullPointerException.
In one of my prod box again started getting NullPointerException. Its taking
more time at commit and throwing an exception. Its trying to commit only 15
records when it is throwing exception. I deleted the index directory manually
and reindexed same data, it just completed in few seconds.
How I can overcome these problems. Thanks in advance!!
Log
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabinversion=2.2} hits=0 status=0 QTime=47
19-Mar-2009 18:55:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabinversion=2.2} status=0 QTime=47
19-Mar-2009 18:57:06 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[27812087, 27812088, 27812089, 27812090, 27812091, 27812079,
27812080, 27812081, ...(15 more)]} 0 122952
19-Mar-2009 18:57:06 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/update
params={wt=javabinversion=2.2} status=0 QTime=122952
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabinversion=2.2} hits=0 status=0 QTime=16
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/admin/ping
params={wt=javabinversion=2.2} status=0 QTime=16
19-Mar-2009 18:59:03 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[27812087, 27812088, 27812089, 27812090, 27812091, 27812079,
27812080, 27812081, ...(15 more)]} 0 141
19-Mar-2009 18:59:03 org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction2-0] webapp=/solr path=/update
params={wt=javabinversion=2.2} status=0 QTime=141
19-Mar-2009 18:59:03 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=false,waitSearcher=false)
StackTrace
java.lang.NullPointerException
at
com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.indexData(AuctionCollectionServiceImpl.java:251)
at
com.fm.search.incrementalindex.service.AuctionCollectionServiceImpl.process(AuctionCollectionServiceImpl.java:135)
at
com.fm.search.job.SearchIndexingJob.executeInternal(SearchIndexingJob.java:68)
at
org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Thanks,
Mahendra
--- On Sat, 3/14/09, Yonik Seeley yo...@lucidimagination.com wrote:
From: Yonik Seeley yo...@lucidimagination.com
Subject: Re: Commit is taking very long time
To: solr-user@lucene.apache.org
Date: Saturday, March 14, 2009, 6:59 AM
From your logs, it looks like the time is spent in closing of the index.
There may be some pending deletes buffered, but they shouldn't take too
long.
There could also be a merge triggered... but this would only happen
sometimes, not every time you commit.
One more relatively recent change in Lucene is to sync the index files
for safety.
Are you perhaps running on Linux with the ext3 filesystem?
Not sure what's causing the null pointer exception... do you have a stack
trace?
-Yonik
http://www.lucidimagination.com
On Fri, Mar 13, 2009 at 9:05 PM, mahendra mahendra
mahendra_featu...@yahoo.com wrote:
Hello,
I am experiencing strange problems while doing commit. I am doing indexing
for every 10 min to update index with data base values. commit is taking 7 to 10
min approximately and my indexing is failing due to null pointer exception. If
first thread is not completed in 10 min the second thread will be starting to
index data.
I changed wait=false for the listener from solrconfig.xml file. It stopped
getting Null pointer exception but the commit is taking 7 to 10 min. I have
approximately 70 to 90 kb of data every time.
listener event=postCommit
class=solr.RunExecutableListener
str
name=exesolr/bin/snapshooter/str
str name=dir./str
bool name=waitfalse/bool
arr name=args strarg1/str
strarg2/str /arr
arr name=env
strMYVAR=val1/str /arr
/listener
I kept all default parameter values in solrconfig.xml except the
ramBuffersize to 512.
Could you please tell me how can I overcome these problems, also some
times I see INFO: Failed to unregister mbean: partitioned because it was
not registered
Mar 13, 2009 11:49:16 AM org.apache.solr.core.JmxMonitoredMap
unregister in my log files.
Log file
ar 13, 2009 1:28:40 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabinwaitFlush=truecommit=truewaitSearcher=trueversion=2.2}
status=0 QTime=247232
Commit is taking very long time
Hello,
I am experiencing strange problems while doing commit. I am doing indexing for
every 10 min to update index with data base values. commit is taking 7 to 10
min approximately and my indexing is failing due to null pointer exception. If
first thread is not completed in 10 min the second thread will be starting to
index data.
I changed wait=false for the listener from solrconfig.xml file. It stopped
getting Null pointer exception but the commit is taking 7 to 10 min. I have
approximately 70 to 90 kb of data every time.
listener event=postCommit class=solr.RunExecutableListener
str name=exesolr/bin/snapshooter/str
str name=dir./str
bool name=waitfalse/bool
arr name=args strarg1/str strarg2/str /arr
arr name=env strMYVAR=val1/str /arr
/listener
I kept all default parameter values in solrconfig.xml except the ramBuffersize
to 512.
Could you please tell me how can I overcome these problems, also some times I
see INFO: Failed to unregister mbean: partitioned because it was not registered
Mar 13, 2009 11:49:16 AM org.apache.solr.core.JmxMonitoredMap unregister in my
log files.
Log file
ar 13, 2009 1:28:40 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabinwaitFlush=truecommit=truewaitSearcher=trueversion=2.2}
status=0 QTime=247232
Mar 13, 2009 1:30:32 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[79827482, 79845504, 79850902, 79850913, 79850697, 79850833,
79850901, 79798207, ...(93 more)]} 0 62578
Mar 13, 2009 1:30:32 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabinversion=2.2} status=0 QTime=62578
Mar 13, 2009 1:30:32 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher init
INFO: Opening searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=63,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=94,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null params={rows=10start=0q=solr}
hits=0 status=0 QTime=0
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null
params={rows=10start=0q=rocks} hits=223 status=0 QTime=0
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null
params={q=static+newSearcher+warming+query+from+solrconfig.xml} hits=4297
status=0 QTime=0
Mar 13, 2009 1:34:38 PM
Re: Commit is taking very long time
From your logs, it looks like the time is spent in closing of the index.
There may be some pending deletes buffered, but they shouldn't take too long.
There could also be a merge triggered... but this would only happen
sometimes, not every time you commit.
One more relatively recent change in Lucene is to sync the index files
for safety.
Are you perhaps running on Linux with the ext3 filesystem?
Not sure what's causing the null pointer exception... do you have a stack trace?
-Yonik
http://www.lucidimagination.com
On Fri, Mar 13, 2009 at 9:05 PM, mahendra mahendra
mahendra_featu...@yahoo.com wrote:
Hello,
I am experiencing strange problems while doing commit. I am doing indexing
for every 10 min to update index with data base values. commit is taking 7 to
10 min approximately and my indexing is failing due to null pointer
exception. If first thread is not completed in 10 min the second thread will
be starting to index data.
I changed wait=false for the listener from solrconfig.xml file. It stopped
getting Null pointer exception but the commit is taking 7 to 10 min. I have
approximately 70 to 90 kb of data every time.
listener event=postCommit class=solr.RunExecutableListener
str name=exesolr/bin/snapshooter/str
str name=dir./str
bool name=waitfalse/bool
arr name=args strarg1/str strarg2/str /arr
arr name=env strMYVAR=val1/str /arr
/listener
I kept all default parameter values in solrconfig.xml except the
ramBuffersize to 512.
Could you please tell me how can I overcome these problems, also some times I
see INFO: Failed to unregister mbean: partitioned because it was not
registered
Mar 13, 2009 11:49:16 AM org.apache.solr.core.JmxMonitoredMap unregister in
my log files.
Log file
ar 13, 2009 1:28:40 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabinwaitFlush=truecommit=truewaitSearcher=trueversion=2.2}
status=0 QTime=247232
Mar 13, 2009 1:30:32 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[79827482, 79845504, 79850902, 79850913, 79850697, 79850833,
79850901, 79798207, ...(93 more)]} 0 62578
Mar 13, 2009 1:30:32 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=/solr path=/update
params={wt=javabinversion=2.2} status=0 QTime=62578
Mar 13, 2009 1:30:32 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher init
INFO: Opening searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=63,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=94,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming searc...@1ba5edf main from searc...@81f25 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for searc...@1ba5edf main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Mar 13, 2009 1:34:38 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to searc...@1ba5edf main
Mar 13, 2009 1:34:38 PM org.apache.solr.core.SolrCore execute
INFO: [EnglishAuction1-0] webapp=null path=null
params={rows=10start=0q=solr} hits=0
