Re: [Dspace-tech] Error in importing Solr stats
Peter: Thank you. I made the setting changes you suggested. I have another question for you and/or the community. When I got the error in converting my dspace log files, I ran the importer several times (since the error said “try again later”). I am wondering if that messed up my statistics. If so, is there a way to clear the stats and redo the importing? George Kozak Digital Library Specialist Cornell University Library Information Technologies (CUL-IT) 501 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 From: pdiet...@gmail.com [mailto:pdiet...@gmail.com] On Behalf Of Peter Dietz Sent: Monday, September 20, 2010 10:55 PM To: George Stanley Kozak Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Error in importing Solr stats Hi George, I'm thinking that this was SOLR having an issue. For some background, check out this thread: http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11062.html The two imbalances currently going on are writing to solr: * too frequently: usage events, which end user traffic hitting your site. possibly 10+ commits per second * too infrequently: statistics importer. possibly too large at potentially millions of records from log importing in a single commit. solr.add(document) is a cheap operation, as is just puts something in queue to be processed later. solr.commit() is an expensive operation, which you want to call just right. Since we don't really know whats right, I suggest we let solr do whats good enough for it, and that would be autoCommit. You can set that up with some defaults: /dspace/solr/statistics/conf/solrconfig.xml maxDocs - number of updates since last commit is greater than this maxTime - oldest uncommited update (in ms) is this long ago autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit If you rely on autoCommit, we can then remove some of the handholding/lockstepping of solr.add(doc), immediately followed by solr.commit() which is terrible for solr performance (on busy instances). So, check out activating autoCommit, and see if things improve. I would suggest that solr autocommit becomes the default for future releases of DSpace, so I'll need some agreement on this. Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Mon, Sep 20, 2010 at 10:28 PM, George Stanley Kozak g...@cornell.edumailto:g...@cornell.edu wrote: Hi... I just migrated my production system to DSpace 1.6.2 from DSpace 1.5.2. Everything went well, until I did the stats-log-importer for the solr statistics. It ran for a long time and then I received this error: About to commit data to solr...Exception: Error opening new searcher. exceeded limit of maxWarmingSearchers=2. Try again later. Does anyone know what caused this error? I didn't see this when I upgraded my test system so I am assuming it has to do with the size of my production logs. IF so, what can I do to fix this? George Kozak Digital Library Specialist Division of Library Information Technologies (DLIT) 501 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.netmailto:DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Error in importing Solr stats
Hi... I just migrated my production system to DSpace 1.6.2 from DSpace 1.5.2. Everything went well, until I did the stats-log-importer for the solr statistics. It ran for a long time and then I received this error: About to commit data to solr...Exception: Error opening new searcher. exceeded limit of maxWarmingSearchers=2. Try again later. Does anyone know what caused this error? I didn't see this when I upgraded my test system so I am assuming it has to do with the size of my production logs. IF so, what can I do to fix this? George Kozak Digital Library Specialist Division of Library Information Technologies (DLIT) 501 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Error in importing Solr stats
Hi George, I'm thinking that this was SOLR having an issue. For some background, check out this thread: http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11062.html The two imbalances currently going on are writing to solr: - too frequently: usage events, which end user traffic hitting your site. possibly 10+ commits per second - too infrequently: statistics importer. possibly too large at potentially millions of records from log importing in a single commit. solr.add(document) is a cheap operation, as is just puts something in queue to be processed later. solr.commit() is an expensive operation, which you want to call just right. Since we don't really know whats right, I suggest we let solr do whats good enough for it, and that would be autoCommit. You can set that up with some defaults: /dspace/solr/statistics/conf/solrconfig.xml maxDocs - number of updates since last commit is greater than this maxTime - oldest uncommited update (in ms) is this long ago autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit If you rely on autoCommit, we can then remove some of the handholding/lockstepping of solr.add(doc), immediately followed by solr.commit() which is terrible for solr performance (on busy instances). So, check out activating autoCommit, and see if things improve. I would suggest that solr autocommit becomes the default for future releases of DSpace, so I'll need some agreement on this. Peter Dietz Systems Developer/Engineer Ohio State University Libraries On Mon, Sep 20, 2010 at 10:28 PM, George Stanley Kozak g...@cornell.eduwrote: Hi... I just migrated my production system to DSpace 1.6.2 from DSpace 1.5.2. Everything went well, until I did the stats-log-importer for the solr statistics. It ran for a long time and then I received this error: About to commit data to solr...Exception: Error opening new searcher. exceeded limit of maxWarmingSearchers=2. Try again later. Does anyone know what caused this error? I didn't see this when I upgraded my test system so I am assuming it has to do with the size of my production logs. IF so, what can I do to fix this? George Kozak Digital Library Specialist Division of Library Information Technologies (DLIT) 501 Olin Library Cornell University Ithaca, NY 14853 607-255-8924 -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
