Hi Eric That sounds like a reasonable approach - however if it is something fairly fundamental about that version of Mulgara and date/time constraints, it's possible that (2) won't help much (as you'd be on the same version of Mulgara).
For your info, the related thread on this (albeit to do with performance rather than memory, but this could still be related as the query's pretty much the same - and a good argument for upgrading in its own right) is: http://fedora-commons.1317035.n2.nabble.com/Fedora-Resource-Index-Query-Serv ice-very-slow-td4144266.html Regards Steve ________________________________________ Steve, Looks like I'm going to have to try a upgrade in some incarnation. While the risearch worked when starting with a fresh proai database as I said in an earlier email, the next time it polled and ran the risearch, the java.langOutOfMemoryError occured, fatally stopping tomcat. To answer your question, here are the tomcat memory settings: /app/jdk1.6.0_14/bin/java -Xms1560m -Xmx1560m -XX:PermSize=256m -XX:MaxPermSize=256m ... So, as you said, it looks like there are 3 options: 1) upgrade to fedora-3.3 2) try a standalone mulgara and configure fedora to use it as per: http://www.fedora-commons.org/confluence/display/FCR30/Resource+Index#Resour ceIndex-RemoteMulgaraDatastoreConfiguration 3) added mulgara and trippi jars to /lib and rebuild index - but this might end up a long search for other dependencies and could break other things I'm likely to go with 2 first then 1, and try to avoid 3. This is all assuming the outOfMemory error will be resolved with the new version of mulgara. Thanks, Eric ________________________________________ From: Steve Bayliss [[email protected]] Sent: Friday, May 14, 2010 4:55 AM To: James, Eric; [email protected] Subject: RE: [Fedora-commons-users] oaiprovider - fedora/risearch - out ofmemory error Hi James Theoretically you could replace the mulgara and trippi jars with those in the 3.3 release, but there might be some other dependencies as well, so I'm not sure if this would work. You could consider setting up a stand-alone instance of Mulgara 2.1.4 (download from http://www.mulgara.org/files/v2.1.4/mulgara-2.1.4-bin.tar.gz) and then: 1) export your current triples from Fedora with something like: wget -O all-rdf.xml http://fedoraAdmin:fedoraad...@localhost:8080/fedora/risearch?type=triples\&; lang=spo\&format=RDF%2FXML\&stream=on\&query=*+*+* or: curl -o all-rdf.xml http://fedoraAdmin:fedoraad...@localhost:8080/fedora/risearch?type=triples\&; lang=spo\&format=RDF%2FXML\&stream=on\&query=*+*+* 2) Load this file into Mulgara 2.1.4 (use the web interface at eg http://localhost:8080/webui/) - note that if you've got Fedora running at the same time you'll have to start Mulgara on a different port to 8080 - check the Mulgara docs on how to do this here http://www.mulgara.org/trac/wiki/StartStop 3) Run the same queries, and see if you encounter any memory problems (you can experiment with the JVM memory settings) At least this might indicate if it's worthwhile upgrading to Fedora 3.3 (or attempting to update the mulgara/trippi jars plus any dependencies). Another alternative would be to configure Fedora to use an external Mulgara triplestore - see the instructions here - http://www.fedora-commons.org/confluence/display/FCR30/Resource+Index#Resour ceIndex-RemoteMulgaraDatastoreConfiguration - but in this instance you should use the same Mulgara version as per the Fedora 3.2 release: check the mulgara jar files to determine what this is. With Mulgara running in a separate JVM you at least have more control over Mulgara's memory. Out of curiosity, what are the JVM memory settings you're currently using for Fedora? Regards Steve > -----Original Message----- > From: James, Eric [mailto:[email protected]] > Sent: 13 May 2010 21:11 > To: Steve Bayliss; [email protected] > Subject: RE: [Fedora-commons-users] oaiprovider - > fedora/risearch - out ofmemory error > > > Thanks Steve. > > We are using fedora3.2. In trying to avoid an upgrade, is it > possible to plug in Mulgara 2.1.4? > > I see in fedora.fcfg: > > <module role="fedora.server.resourceIndex.ResourceIndex" > class="fedora.server.resourceIndex.ResourceIndexModule"> > ... > <param name="datastore" value="localMulgaraTriplestore"> > > and > > <datastore id="localMulgaraTriplestore"> > ... > <param name="connectorClassName" > value="org.trippi.impl.mulgara.MulgaraConnector"> > > Eric > > ________________________________________ > From: Steve Bayliss [[email protected]] > Sent: Thursday, May 13, 2010 12:41 PM > To: James, Eric; [email protected] > Subject: RE: [Fedora-commons-users] oaiprovider - > fedora/risearch - out ofmemory error > > Hi Eric > > What version of Fedora are you using? I ask as in version > 3.3 we upgraded > to Mulgara 2.1.4; in the previous version of Fedora/Mulgara there were > reports of some query issues particularly related to > before/after dateTime > constraints using the #xsd model. > > Regards > Steve > > > -----Original Message----- > > From: James, Eric [mailto:[email protected]] > > Sent: 13 May 2010 17:20 > > To: [email protected] > > Subject: [Fedora-commons-users] oaiprovider - fedora/risearch > > - out ofmemory error > > > > > > Hi, > > > > I recently made ~1100 objects in my fedora repository visible > > to the oaiprovider webapp by adding an itemID to these > > objects RELS-EXT. I successfully tested this with 2 objects > > individually. But after making the batch change, the > > oaiprovider failed with a java.lang.OutOfMemoryError. > > > > The route cause of the OutOfMemoryError is a fedora risearch, > > particularly the query: > > > > > > select $item $itemID $date $state > > > > from <#ri> > > > > where $item <http://www.openarchives.org/OAI/2.0/itemID> $itemID > > > > and $item <info:fedora/fedora-system:def/model#state> $state > > > > and $item <info:fedora/fedora-system:def/view#disseminates> $diss > > > > and $diss > > <info:fedora/fedora-system:def/view#disseminationType> > > <info:fedora/*/DC> > > > > and $item > <info:fedora/fedora-system:def/view#lastModifiedDate> $date > > > > and $date <http://mulgara.org/mulgara#after> > > '2010-05-12T19:37:55.289Z'^^<http://www.w3.org/2001/XMLSchema# > > dateTime> in <#xsd> > > > > and $date <http://mulgara.org/mulgara#before> > > '2010-05-13T13:07:04.746Z'^^<http://www.w3.org/2001/XMLSchema# > > dateTime> in <#xsd> > > > > > > > > But if I run the same search without the mulgara before and > > after conditions it works fine: > > > > > > > > select $item $itemID $date $state > > > > from <#ri> > > > > where $item <http://www.openarchives.org/OAI/2.0/itemID> $itemID > > > > and $item <info:fedora/fedora-system:def/model#state> $state > > > > and $item <info:fedora/fedora-system:def/view#disseminates> $diss > > > > and $diss > > <info:fedora/fedora-system:def/view#disseminationType> > > <info:fedora/*/DC> > > > > and $item > <info:fedora/fedora-system:def/view#lastModifiedDate> $date > > > > > > > > I was able to workaround this by refreshing the database, > > clearing the /cache /sessions /schemas directories, and > > restarting the oaiprovider. But I'm concerned this issue > > will return again as I make available ~4000 more objects for > > harvesting, and don't wan't to have to keep clean slating the > > harvester every time new objects are added. > > > > > > > > One thing I tried was the mySQLResultTrickling parameter, but > > I think that is related to mysql, not the triplestore: > > > > proai.db.mySQLResultTrickling = true > > > > > > > > Is this a known bug? Does anyone have any insight, solution > > to this problem? > > > > > > > > Below is the outOfMemory stack trace: > > > > > > > > DEBUG 2010-05-12 17:05:24.693 [Thread-7324] (FedoraClient) > > FedoraClient is getting > > http://localhost:8083/fedora/risearch?query=select+%24item+%24 > > itemID+%24date+%24state%0Afrom+++%3C%23ri%3E%0Awhere++%2 > > > > 4item+++++++++++%3Chttp%3A%2F%2Fwww.openarchives.org%2FOAI%2F2 > > .0%2FitemID%3E+%24itemID%0Aand++++%24item+++++%3Cinfo%3Afedora > > %2Ffedora-system%3Adef%2Fmodel%23state%3E+%24state%0Aand+%24it > > em+%3Cinfo%3Afe > > > > dora%2Ffedora-system%3Adef%2Fview%23disseminates%3E+%24diss%0A > > and+%24diss+%3Cinfo%3Afedora%2Ffedora-system%3Adef%2Fview%23di > > sseminationType%3E+%3Cinfo%3Afedora%2F*%2FDC%3E%0Aand++++%24it > > em+++++%3Cinfo% > > > > 3Afedora%2Ffedora-system%3Adef%2Fview%23lastModifiedDate%3E+%2 > > 4date%0Aand++++%24date+++++++++++%3Chttp%3A%2F%2Fmulgara.org%2 > > Fmulgara%23after%3E+%272010-05-12T19%3A37%3A55.289Z%27%5E%5E%3 > > Chttp%3A%2F%2Fw > > > > ww.w3.org%2F2001%2FXMLSchema%23dateTime%3E+in+%3C%23xsd%3E%0Aa > > nd++++%24date+++++++++++%3Chttp%3A%2F%2Fmulgara.org%2Fmulgara% > > 23before%3E+%272010-05-12T21%3A05%3A24.481Z%27%5E%5E%3Chttp%3A > > %2F%2Fwww.w3.or > > > > g%2F2001%2FXMLSchema%23dateTime%3E+in+%3C%23xsd%3E%0Aorder++by > > +%24itemID+asc&format=Sparql&type=tuples&lang=itql > > > > mmap failed for CEN and END part of zip file > > > > ERROR 2010-05-12 17:05:29.403 [http-8083-Processor23] > > ([RISearchServlet]) Servlet.service() for servlet > > RISearchServlet threw exception > > > > java.lang.OutOfMemoryError > > > > at java.util.zip.ZipFile.open(Native Method) > > > > at java.util.zip.ZipFile.<init>(ZipFile.java:114) > > > > at java.util.jar.JarFile.<init>(JarFile.java:133) > > > > at java.util.jar.JarFile.<init>(JarFile.java:97) > > > > at > > org.apache.catalina.loader.WebappClassLoader.openJARs(WebappCl > > assLoader.java:1766) > > > > at > > org.apache.catalina.loader.WebappClassLoader.findResourceInter > > nal(WebappClassLoader.java:1996) > > > > at > > org.apache.catalina.loader.WebappClassLoader.findClassInternal > > (WebappClassLoader.java:1796) > > > > at > > org.apache.catalina.loader.WebappClassLoader.findClass(WebappC > > lassLoader.java:875) > > > > at > > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappC > > lassLoader.java:1330) > > > > at > > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappC > > lassLoader.java:1209) > > > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > > > > at > > > org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:755) > > > > at > > org.mulgara.resolver.DatabaseSession.query(DatabaseSession.java:465) > > > > at > > > org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:129) > > > > at > > org.trippi.impl.base.ConcurrentTriplestoreReader.findTuples(Co > ncurrentTriplestoreReader.java:79) > > > > at > > fedora.server.resourceIndex.ResourceIndexImpl.findTuples(Resou > > rceIndexImpl.java:279) > > > > at > > fedora.server.resourceIndex.ResourceIndexModule.findTuples(Res > > ourceIndexModule.java:296) > > > > at org.trippi.server.TrippiServer.find(TrippiServer.java:119) > > > > at > org.trippi.server.http.TrippiServlet.doFind(TrippiServlet.java:512) > > > > at > org.trippi.server.http.TrippiServlet.doGet(TrippiServlet.java:377) > > > > at > > fedora.server.access.RISearchServlet.doGet(RISearchServlet.java:103) > > > > at > org.trippi.server.http.TrippiServlet.doGet(TrippiServlet.java:269) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > er(ApplicationFilterChain.java:269) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > cationFilterChain.java:188) > > > > at > > fedora.server.security.servletfilters.FilterSetup.doFilter(Fil > > terSetup.java:234) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > er(ApplicationFilterChain.java:215) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > cationFilterChain.java:188) > > > > at > > fedora.server.security.servletfilters.FilterSetup.doFilter(Fil > > terSetup.java:234) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > er(ApplicationFilterChain.java:215) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > cationFilterChain.java:188) > > > > at > > fedora.server.security.servletfilters.FilterSetup.doFilter(Fil > > terSetup.java:234) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > er(ApplicationFilterChain.java:215) > > > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > cationFilterChain.java:188) > > > > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > rapperValve.java:213) > > > > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > ontextValve.java:174) > > > > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Aut > > henticatorBase.java:525) > > > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > > Valve.java:127) > > > > > > > > > > > > Thanks, > > > > Eric > > > > > > > > > > > > -------------------------------------------------------------- > > ---------------- > > > > _______________________________________________ > > Fedora-commons-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > > > ------------------------------------------------------------------------------ _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
