Hi Christian Yes, I think that is most likely the cause.
Regards, Gert On 03/05/2012, at 11.39, Christian Rohrer wrote: > Hi Gert > > On 27.04.2012, at 17:47, Gert Schmeltz Pedersen wrote: > >> Hi Remi, >> >> The variable time after a restart before the problems start indicates that >> it would be best to look at the operation(s) occurring just before the >> problems start, what does the log say there? I think that the gfindObjects >> operations, that you list log lines for, could hardly be responsible. >> >> The requests from search engines, some with the old sort parameter AUTO? You >> have to explain how these relate to the problem. >> >> I see in your code that you have renamed "fedoragsearch" to "indexer". Are >> you sure you have done this consistently? You may need to check carefully >> the fgsconfigFinal files. The problems with segmentation files and locks >> indicate that your configuration is not consistent. >> > > Thanks for your hint, I now switched it all back to fedoragsearch and the > name of the index to FgsIndex, but I still got the problem. The locking > itself occurs, when I update objects, which causes Fedora to create a lot of > update messages. This in turn causes GSearch to lock up eventually. > > In the meantime I found another possible reason for the locking behaviour: I > had both update mechanisms enabled, ie. Messaging and via REST. I have now > disabled the update via REST and I haven't had any further locking errors > since. Do you think this could be the cause for my troubles? > > Regards > christian > > >> Best >> Gert >> >> >> On 26/04/2012, at 13.02, Christian Rohrer wrote: >> >>> >>> Hi Gert >>> >>> sorry for the late rely, I subscribed this list in digest-mode, and your >>> message only got delivered today. >>> >>> >>>> >>>>> From: Gert Schmeltz Pedersen <gerts...@gmail.com> >>>>> Subject: Re: [fcrepo-dev] FedoraGSearch locks and hogs file connections >>>>> Date: 20. April 2012 10:46:11 MESZ >>>>> To: fedora-commons-developers@lists.sourceforge.net >>>>> Reply-To: fedora-commons-developers@lists.sourceforge.net >>>>> >>>>> Hi Christian, >>>>> >>>>> Could you give a little more information, please. After a restart of >>>>> tomcat, how much activity has happened before problems start? and what >>>>> was that activity? >>>>> >>> >>> The time after a restart before the problem occurs ist quite variable. From >>> a few minutes to several days. >>> >>> Activities are request like this: >>> INFO 2012-04-26 00:17:02,702 (RESTImpl) >>> request=query=rels-ext.isMemberOfCollection%3A%22LOR%3A1%22+AND+fedora-model.hasModel%3A%22LORmodel%3Aobject%22&hitPageSize=5&hitPageStart=1&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse&operation=gfindObjects&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml& >>> timeusedms=83 >>> INFO 2012-04-26 00:17:02,819 (RESTImpl) >>> request=query=rels-ext.isMemberOfCollection%3A%22LOR%3A1%22+AND+fedora-model.hasModel%3A%22LORmodel%3Afolder%22&operation=gfindObjects&hitPageSize=1000&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse& >>> timeusedms=101 >>> INFO 2012-04-26 00:17:06,023 (RESTImpl) >>> request=query=dcterms.source%3A%22LOR%3A5806%22+or+PID%3A%22LOR%3A5806%22&operation=gfindObjects&hitPageSize=1000&indexName=ChorIndex&restXslt=copyXml&resultPageXslt=copyXml&sortFields=PID%2CSCORE%2Cfalse%3Bfgs.label%2Cde-CH%2Cfalse& >>> remoteUser= >>> FEDORA_AUX_SUBJECT_ATTRIBUTES= >>> >>> Among them are a lot of requests from search engines (Google, Bing, etc). >>> >>> Some of them somehow still use the old sort parameter AUTO which results in >>> an error: >>> FEDORA_AUX_SUBJECT_ATTRIBUTES= >>> ERROR 2012-04-26 00:17:09,997 (RESTImpl) >>> dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 26 >>> 00:17:09 CEST 2012 gfindObjects executeQuery error: >>> dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 26 >>> 00:17:09 CEST 2012 getHits >>> sortFields='fgs.lastModifiedDate,AUTO,false;PID,SCORE' : unknown sortType >>> string 'AUTO' in 'fgs.lastModifiedDate,AUTO,false' >>> >>> If you have time, you can have a look at the code I use to compile my >>> war-file at >>> https://github.com/hydrat/gsearch >>> >>> >>> Thanks for your help >>> christian >>> >>> >>> >>> >>>>> Gert >>>>> >>>>> >>>>> On 19/04/2012, at 09.05, Christian Rohrer wrote: >>>>> >>>>>> Hi list >>>>>> >>>>>> I have recently upgraded our Fedora installation to 3.5 and GSearch >>>>>> (using Lucene) to 2.4.1. >>>>>> >>>>>> Since this upgrade GSearch causes us problems. It keeps opening >>>>>> connections to the segmentation files, which remain open until the OS >>>>>> limit is reached, after which a restart of Tomcat is needed in order to >>>>>> close all open connection of GSearch. Has anybody else observed this >>>>>> behavior? >>>>>> >>>>>> The other unusual thing (might be related?) are locks that throw error >>>>>> messages in the form: >>>>>> >>>>>> dk.defxws.fedoragsearch.server.errors.GenericSearchException: Thu Apr 19 >>>>>> 09:02:45 CEST 2012 IndexWriter new error indexName=ChorIndex : >>>>>> ; nested exception is: >>>>>> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed >>>>>> out: NativeFSLock@/opt/fedora35/indexer/ChorIndex/write.lock >>>>>> at >>>>>> dk.defxws.fgslucene.OperationsImpl.getIndexWriter(OperationsImpl.java:695) >>>>>> at >>>>>> dk.defxws.fgslucene.OperationsImpl.browseIndex(OperationsImpl.java:151) >>>>>> at >>>>>> dk.defxws.fedoragsearch.server.RESTImpl.browseIndex(RESTImpl.java:267) >>>>>> at dk.defxws.fedoragsearch.server.RESTImpl.doGet(RESTImpl.java:140) >>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >>>>>> at >>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) >>>>>> at >>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >>>>>> at >>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >>>>>> at >>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >>>>>> at >>>>>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:316) >>>>>> at >>>>>> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) >>>>>> at >>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >>>>>> at >>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >>>>>> at >>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >>>>>> at >>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) >>>>>> at >>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) >>>>>> at >>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) >>>>>> at >>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) >>>>>> at java.lang.Thread.run(Thread.java:636) >>>>>> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock >>>>>> obtain timed out: NativeFSLock@/opt/fedora35/indexer/ChorIndex/write.lock >>>>>> at org.apache.lucene.store.Lock.obtain(Lock.java:84) >>>>>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108) >>>>>> at >>>>>> dk.defxws.fgslucene.OperationsImpl.getIndexWriter(OperationsImpl.java:693) >>>>>> ... 19 more >>>>>> >>>>>> >>>>>> Any ideas what could be the cause for this? >>>>>> >>>>>> Best regards >>>>>> christian >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> For Developers, A Lot Can Happen In A Second. >>>>>> Boundary is the first to Know...and Tell You. >>>>>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! >>>>>> http://p.sf.net/sfu/Boundary-d2dvs2_______________________________________________ >>>>>> Fedora-commons-developers mailing list >>>>>> Fedora-commons-developers@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> For Developers, A Lot Can Happen In A Second. >>>>> Boundary is the first to Know...and Tell You. >>>>> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! >>>>> http://p.sf.net/sfu/Boundary-d2dvs2 >>>>> _______________________________________________ >>>>> Fedora-commons-developers mailing list >>>>> Fedora-commons-developers@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >>>> >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. Discussions >>> will include endpoint security, mobile security and the latest in malware >>> threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ >>> Fedora-commons-developers mailing list >>> Fedora-commons-developers@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Fedora-commons-developers mailing list >> Fedora-commons-developers@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers > > -- > SWITCH > Serving Swiss Universities > -------------------------- > Christian Rohrer, Software Engineer > Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland > phone +41 44 268 15 79, fax +41 44 268 15 68 > christian.roh...@switch.ch, http://www.switch.ch > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ > Fedora-commons-developers mailing list > Fedora-commons-developers@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Fedora-commons-developers mailing list Fedora-commons-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fedora-commons-developers
