Peter:
Thank you. I made the setting changes you suggested. I have another question
for you and/or the community. When I got the error in converting my dspace log
files, I ran the importer several times (since the error said “try again
later”). I am wondering if that messed up my statistics. If so, is there a
way to clear the stats and redo the importing?
George Kozak
Digital Library Specialist
Cornell University Library Information Technologies (CUL-IT)
501 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
From: pdiet...@gmail.com [mailto:pdiet...@gmail.com] On Behalf Of Peter Dietz
Sent: Monday, September 20, 2010 10:55 PM
To: George Stanley Kozak
Cc: dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] Error in importing Solr stats
Hi George,
I'm thinking that this was SOLR having an issue.
For some background, check out this thread:
http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11062.html
The two imbalances currently going on are writing to solr:
* too frequently: usage events, which end user traffic hitting your site.
possibly 10+ commits per second
* too infrequently: statistics importer. possibly too large at potentially
millions of records from log importing in a single commit.
solr.add(document) is a cheap operation, as is just puts something in queue to
be processed later.
solr.commit() is an expensive operation, which you want to call "just right".
Since we don't really know whats right, I suggest we let solr do whats good
enough for it, and that would be autoCommit.
You can set that up with some defaults:
/dspace/solr/statistics/conf/solrconfig.xml
maxDocs - number of updates since last commit is greater than this
maxTime - oldest uncommited update (in ms) is this long ago
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>1000</maxTime>
</autoCommit>
If you rely on autoCommit, we can then remove some of the
handholding/lockstepping of solr.add(doc), immediately followed by
solr.commit() which is terrible for solr performance (on busy instances).
So, check out activating autoCommit, and see if things improve. I would suggest
that solr autocommit becomes the default for future releases of DSpace, so I'll
need some agreement on this.
Peter Dietz
Systems Developer/Engineer
Ohio State University Libraries
On Mon, Sep 20, 2010 at 10:28 PM, George Stanley Kozak
<g...@cornell.edu<mailto:g...@cornell.edu>> wrote:
Hi...
I just migrated my production system to DSpace 1.6.2 from DSpace 1.5.2.
Everything went well, until I did the stats-log-importer for the solr
statistics.
It ran for a long time and then I received this error:
About to commit data to solr...Exception: Error opening new searcher. exceeded
limit of maxWarmingSearchers=2. Try again later.
Does anyone know what caused this error? I didn't see this when I upgraded my
test system so I am assuming it has to do with the size of my production logs.
IF so, what can I do to fix this?
George Kozak
Digital Library Specialist
Division of Library Information Technologies (DLIT)
501 Olin Library
Cornell University
Ithaca, NY 14853
607-255-8924
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net<mailto:DSpace-tech@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/dspace-tech
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
