Hi Remi, The variable time after a restart before the problems start indicates that it would be best to look at the operation(s) occurring just before the problems start, what does the log say there? I think that the gfindObjects operations, that you list log lines for, could hardly be responsible.
The requests from search engines, some with the old sort parameter AUTO? You have to explain how these relate to the problem. I see in your code that you have renamed "fedoragsearch" to "indexer". Are you sure you have done this consistently? You may need to check carefully the fgsconfigFinal files. The problems with segmentation files and locks indicate that your configuration is not consistent. Best Gert On 26/04/2012, at 13.02, Christian Rohrer wrote: > > Hi Gert > > sorry for the late rely, I subscribed this list in digest-mode, and your > message only got delivered today. > > >> >>> From: Gert Schmeltz Pedersen <gerts...@gmail.com> >>> Subject: Re: [fcrepo-dev] FedoraGSearch locks and hogs file connections >>> Date: 20. April 2012 10:46:11 MESZ >>> To: fedora-commons-developers@lists.sourceforge.net >>> Reply-To: fedora-commons-developers@lists.sourceforge.net >>> >>> Hi Christian, >>> >>> Could you give a little more information, please. After a restart of >>> tomcat, how much activity has happened before problems start? and what was >>> that activity? >>> > > The time after a restart before the problem occurs ist quite variable. From a > few minutes to several days. > > Activities are request like this: > INFO 2012-04-26 00:17:02,702 (RESTImpl) > request=query=rels-ext.isMemberOfCollection%3A%22LOR%3A1%22+AND+fedora-model.hasModel%3A%22LORmodel%3Aobject%22&hitPageSize=5&hitPageStart=1&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse&operation=gfindObjects&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml& > timeusedms=83 > INFO 2012-04-26 00:17:02,819 (RESTImpl) > request=query=rels-ext.isMemberOfCollection%3A%22LOR%3A1%22+AND+fedora-model.hasModel%3A%22LORmodel%3Afolder%22&operation=gfindObjects&hitPageSize=1000&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse& > timeusedms=101 > INFO 2012-04-26 00:17:06,023 (RESTImpl) > request=query=dcterms.source%3A%22LOR%3A5806%22+or+PID%3A%22LOR%3A5806%22&operation=gfindObjects&hitPageSize=1000&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse& > remoteUser= > FEDORA_AUX_SUBJECT_ATTRIBUTES= > > Among them are a lot of requests from search engines (Google, Bing, etc). > > Some of them somehow still use the old sort parameter AUTO which results in > an error: > FEDORA_AUX_SUBJECT_ATTRIBUTES= > ERROR 2012-04-26 00:17:09,997 (RESTImpl) > dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 26 > 00:17:09 CEST 2012 gfindObjects executeQuery error: > dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 26 > 00:17:09 CEST 2012 getHits > sortFields='fgs.lastModifiedDate,AUTO,false;PID,SCORE' : unknown sortType > string 'AUTO' in 'fgs.lastModifiedDate,AUTO,false' > > If you have time, you can have a look at the code I use to compile my > war-file at > https://github.com/hydrat/gsearch > > > Thanks for your help > christian > > > > >>> Gert >>> >>> >>> On 19/04/2012, at 09.05, Christian Rohrer wrote: >>> >>>> Hi list >>>> >>>> I have recently upgraded our Fedora installation to 3.5 and GSearch (using >>>> Lucene) to 2.4.1. >>>> >>>> Since this upgrade GSearch causes us problems. It keeps opening >>>> connections to the segmentation files, which remain open until the OS >>>> limit is reached, after which a restart of Tomcat is needed in order to >>>> close all open connection of GSearch. Has anybody else observed this >>>> behavior? >>>> >>>> The other unusual thing (might be related?) are locks that throw error >>>> messages in the form: >>>> >>>> dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 19 >>>> 09:02:45 CEST 2012 IndexWriter new error indexName=ChorIndex : >>>> ; nested exception is: >>>> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed >>>> out: NativeFSLock@/opt/fedora35/indexer/ChorIndex/write.lock >>>> at >>>> dk.defxws.fgslucene.OperationsImpl.getIndexWriter(OperationsImpl.java:695) >>>> at >>>> dk.defxws.fgslucene.OperationsImpl.browseIndex(OperationsImpl.java:151) >>>> at >>>> dk.defxws.fedoragsearch.server.RESTImpl.browseIndex(RESTImpl.java:267) >>>> at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:140) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >>>> at >>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >>>> at >>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >>>> at >>>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:316) >>>> at >>>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) >>>> at >>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >>>> at >>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >>>> at >>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >>>> at >>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) >>>> at >>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) >>>> at >>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) >>>> at >>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) >>>> at java.lang.Thread.run(Thread.java:636) >>>> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain >>>> timed out: NativeFSLock@/opt/fedora35/indexer/ChorIndex/write.lock >>>> at org.apache.lucene.store.Lock.obtain(Lock.java:84) >>>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108) >>>> at >>>> dk.defxws.fgslucene.OperationsImpl.getIndexWriter(OperationsImpl.java:693) >>>> ... 19 more >>>> >>>> >>>> Any ideas what could be the cause for this? >>>> >>>> Best regards >>>> christian >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> For Developers, A Lot Can Happen In A Second. >>>> Boundary is the first to Know...and Tell You. >>>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! >>>> http://p.sf.net/sfu/Boundary-d2dvs2_______________________________________________ >>>> Fedora-commons-developers mailing list >>>> Fedora-commons-developers@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>> >>> >>> ------------------------------------------------------------------------------ >>> For Developers, A Lot Can Happen In A Second. >>> Boundary is the first to Know...and Tell You. >>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! >>> http://p.sf.net/sfu/Boundary-d2dvs2 >>> _______________________________________________ >>> Fedora-commons-developers mailing list >>> Fedora-commons-developers@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >> > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ > Fedora-commons-developers mailing list > Fedora-commons-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Fedora-commons-developers mailing list Fedora-commons-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
