Hi Dave, Thanks for helping me and particularly for your prompt and valuable response.
I understand and agree with the gc() explanation. Thanks. Memory limit was my best theory about what is going on with my application. Namely, I am trying to migrate an application using Jena from dedicated server to a shared server. The exactly same code is running without any problems until it is ported on the dedicated server. I paste my error trace at the end of the message for your reference. The application simply diminish without any exception caught. I traced it down to the single line instantiating OntModel (default reasoner, no ontology loaded initially): OntModel metaMCQ = null; metaMCQ = ModelFactory.createOntologyModel(); Exactly the same code works on the dedicated server while keeps crashing on the shared server? Thank you in advance, Milorad My error trace follows: java.lang.ExceptionInInitializerError at com.hp.hpl.jena.rdf.model.impl.RDFReaderFImpl.<clinit>(RDFReaderFImpl.java:85) at com.hp.hpl.jena.rdf.model.impl.ModelCom.<clinit>(ModelCom.java:42) at com.hp.hpl.jena.rdf.model.ModelFactory.createDefaultModel(ModelFactory.java:122) at com.hp.hpl.jena.rdf.model.ModelFactory.createDefaultModel(ModelFactory.java:116) at com.hp.hpl.jena.vocabulary.OWL.<clinit>(OWL.java:37) at com.hp.hpl.jena.ontology.ProfileRegistry.<clinit>(ProfileRegistry.java:48) at com.hp.hpl.jena.rdf.model.ModelFactory.createOntologyModel(ModelFactory.java:344) at com.virtuona.service.semcqgenerator.SeMCQgeneratorService.runApp(SeMCQgeneratorService.java:182) at com.virtuona.service.semcqgenerator.SeMCQgeneratorService.runService(SeMCQgeneratorService.java:380) at com.virtuona.service.ServiceCaller.callService(ServiceCaller.java:28) at com.virtuona.service.ServletServiceCaller.callService(ServletServiceCaller.java:397) at com.virtuona.service.ServletServiceCaller.procesRequest(ServletServiceCaller.java:258) at com.virtuona.service.ServletServiceCaller.doPost(ServletServiceCaller.java:107) at javax.servlet.http.HttpServlet.service(HttpServlet.java:165) at javax.servlet.http.HttpServlet.service(HttpServlet.java:103) at com.caucho.server.http.FilterChainServlet.doFilter(FilterChainServlet.java:96) at com.caucho.server.http.Invocation.service(Invocation.java:315) at com.caucho.server.http.CacheInvocation.service(CacheInvocation.java:135) at com.caucho.server.http.RunnerRequest.handleRequest(RunnerRequest.java:346) at com.caucho.server.http.RunnerRequest.handleConnection(RunnerRequest.java:274) at com.caucho.server.TcpConnection.run(TcpConnection.java:139) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.NullPointerException at java.util.XMLUtils.load(XMLUtils.java:62) at java.util.Properties.loadFromXML(Properties.java:701) at com.hp.hpl.jena.util.Metadata.read(Metadata.java:67) at com.hp.hpl.jena.util.Metadata.addMetadata(Metadata.java:41) at com.hp.hpl.jena.util.Metadata.<init>(Metadata.java:35) at com.hp.hpl.jena.JenaRuntime.<clinit>(JenaRuntime.java:25) ... 22 more >________________________________ > From: Dave Reynolds <[email protected]> >To: Milorad Tosic <[email protected]> >Cc: "[email protected]" <[email protected]> >Sent: Friday, December 16, 2011 4:41 PM >Subject: Re: comparison of JVM and Jena versions by memory usage > >Hi, > >On Fri, 2011-12-16 at 05:39 -0800, Milorad Tosic wrote: >> Hi, >> >> >> There is a short discussion on the former Jena Development/Support mailing >> list related to memory usage of Jena OntModel using different versions of >> Jena. For your information I have included a segment of the correspondence >> at the end of the message. Since I had a problem that looks much like memory >> usage limit overrun, I got interested in the problem. >> >> >> I repeated the same experiment but with >> different conclusions? I compared Jena 2.6.0 with Jena 2.6.4. I couldn't >> notice the difference between Jena versions as reported in your >> previous experiment (though I used the same code). Tried with several >> variations of the OntModel creations styles but with minor differences >> in memory usage only (within 10%). I run the experiment from within >> Eclipse with JVM set to jre6 using compiler compliance level 1.5 (as >> well as 1.6). The total memory usage was around 27975k. >> >> Than, I changed the JVM to jdk 1.5. There was similar relative >> difference between the Jena 2.6.0 and Jena 2.6.4 (within 10%). However, >> the absolute amount of used memory dropped to about 3400k. >> >> Than I again changed the JVM to jdk 1.6. As in all previous testing >> configurations, there was similar relative difference between the Jena >> 2.6.0 and Jena 2.6.4 (within 10%). However, the absolute amount of used >> memory was about 6501k. >> >> Consequently, I have two problems now: >> >> 1) I couldn't repeat you experiment with the same results. What I am doing >> wrong? > >Assuming you are running the code that Rick originally posted then the >problem is that that uses a naive way to measure memory usage. >If I run with java 1.6 (sun-java on Linux) and the current Jena snapshot >and run in eclipse I get: > >time: 19, totMem: 122688k, usedMem: 4859k > >If I run identical code from the command line I get: >time: 12, totMem: 122688k, usedMem: 28548k > >The difference is down to whether a garbage collect has kicked in. If I >force a System.gc() in that code then I get: > >time: 86, totMem: 122688k, usedMem: 4006k > >where the extra time comes from the garbage collect. > > >> 2) What should I do to put the memory usage under my control? Namely, 5 >> times larger memory usage cause my application to broke on machines >> with limited memory usage. > >I don't think there's any evidence that the problem in the thread that >you are referring to is still an issue. > >You would need to explain more about what you are actually do in your >application before we could help with that. However, if you are >currently using inference and are having memory problems then avoiding >inference, or using a weaker inference setting, would be a good place to >start. > >Dave > >> >> Thanks, >> Milorad Tosic >> > >> > --- In [email protected], Dave Reynolds <dave.e.reynolds@...> wrote: >> > > >> > > Rick, >> > > >> > > Thanks for letting us know, thanks for the clear report. >> > > >> > > Dave >> > > >> > > rspates2 wrote: >> > > > Dave, >> > > > >> > >> > > I grabbed OWLProfile.java and reran my tests. It looks fixed >> to me in terms of speed and memory use. We should be able to use 2.6.2 >> now. >> > > > >> > > > Thanks for your swift resolution to the problem. >> > > > >> > > > Rick >> > > > >> > > > --- In [email protected], Dave Reynolds <dave.e.reynolds@> >> > > > wrote: >> > > >> Hi Rick, >> > > >> >> > > >> A follow up on my earlier off-list response. >> > > >> >> > > >> Thanks again for giving us a test case for this. It turns out not to >> > > >> be >> > > >> a reasoner issue as such. There was a change in the internals of the >> > > >> Ontology API which worked inefficiently when a reasoner is present. >> > > >> Ian >> > > >> has checked in some patches for this which brings the memory use down >> > > >> a >> > > >> lot and gives a 5x speed improvement. >> > > >> >> > > >> It is not quite back to 2.6.0 timings but is a significant >> > > >> improvement. >> > > >> >> > > >> We'll investigate a little more but this may be as close as we can >> > > >> get >> > > >> for now. Let us know if this is enough to make it usable for you. >> > > >> >> > > >> Cheers, >> > > >> Dave >> > > >> >> > > >> rspates2 wrote: >> > > >>> Dave, >> > > >>> >> > > >>> The difference seems to be between 2.6.0 and 2.6.2, >> > > >>> >> > > >>> OntModel.listClasses(): >> > > >>> 2.6.2 time: 2187, totMem: 19948k, usedMem: 13345k >> > > >>> 2.6.0 time: 110, totMem: 5056k, usedMem: 3681k >> > > >>> 2.5.7 time: 78, totMem: 5056k, usedMem: 3726k >> > > >>> >> > > >>> listHierarchyRootClasses(): >> > > >>> 2.6.2 time: 2312, totMem: 20152k, usedMem: 13859k >> > > >>> 2.6.0 time: 172, totMem: 5056k, usedMem: 4312k >> > > >>> 2.5.7 time: 188, totMem: 5056k, usedMem: 4078k >> > > >>> >> > > >>> Let me know if you need anything else. >> > > >>> >> > > >>> Thanks, >> > > >>> Rick >> > > >>> >> > > >>> --- In [email protected], Dave Reynolds <dave.e.reynolds@> >> > > >>> wrote: >> > > >>>> rspates2 wrote: >> > > >>>>> Hi, >> > > >>>>> >> > >> > >>>>> We switched our application from Jena 2.5.7 >> to 2.6.2. For an operation on OntModel.listHierarchyRootClasses() with >> an OntModelSpec of OWL_MEM_RDFS_INF, we saw a very significant increase >> in the use of memory and processing time, including out-of-memory >> condition for our full ontology. Using a subset of our ontology and the >> code below to demonstrate, we measured processing times and memory use: >> > > >>>>> >> > > >>>>> 2.6.2 time: 2063, totMem: 20008k, usedMem: 14539k >> > > >>>>> 2.5.7 time: 172, totMem: 5056k, usedMem: 3974k >> > > >>>> Certainly a huge jump :( >> > > >>>> >> > >> > >>>>> The specifications OWL_MEM_RDFS_INF and >> OWL_DL_MEM_RDFS_INF produced comparable results, while >> OWL_DL_MEM_TRANS_INF at 2.6.2 was comparable to our 2.5.7 run with >> OWL_MEM_RDFS_INF. >> > > >>>>> >> > > >> >>>>> I can post the subset ontology if you need it (170 >> k), but thought there might be some known explanation you could provide >> without running the demo code. >> > > >>>>> >> > > >>>>> Any idea what is causing this? >> > > >>>> No immediate ideas. The RFDS rule file itself hasn't changed in >> > > >>>> that >> > > >>>> time frame. It is tricky to trace back code changes over that >> > > >>>> period >> > > >>>> because we did a code reorg which hides a lot of the history. >> > > >>>> However, >> > > >>>> from memory (and the release notes) I can't think of anything that >> > > >>>> should have had an effect like that. There were changes around >> > > >>>> handling >> > > >>>> validation of data types but that isn't relevant here. We >> > > >>>> generalized >> > > >>>> handling of non-legal RDF triples but again that shouldn't be >> > > >>>> relevant. >> > > >>>> >> > > >>>> Could you try two experiments? >> > > >>>> >> > > >>>> First, would it be possible to time a simpler operation, such as >> > > >>>> listing >> > > >>>> all classes, to eliminate listHierarchyRootClasses as a factor? >> > > >>>> >> > > >>>> Secondly, is it possible to check with Jena 2.6.0 - that would >> > > >>>> separate >> > > >>>> the Java 5 changes and the datatype validation from more recent >> > > >>>> tweaks? >> > > >>>> >> > > >>>> If that's a problem then send me your ontology subset and I'll take >> > > >>>> a >> > > >>>> look when I can. >> > > >>>> >> > > >>>> Dave >> > > >>>> >> > > >>>>> Thanks for your help, >> > > >>>>> Rick >> > > >>>>> >> > > >>>>> import java.io.InputStream; >> > > >>>>> import org.junit.Test; >> > > >>>>> >> > > >>>>> import com.hp.hpl.jena.ontology.OntModel; >> > > >>>>> import com.hp.hpl.jena.ontology.OntModelSpec; >> > > >>>>> import com.hp.hpl.jena.rdf.model.Model; >> > > >>>>> import com.hp.hpl.jena.rdf.model.ModelFactory; >> > > >>>>> import com.hp.hpl.jena.util.iterator.ExtendedIterator; >> > > >>>>> >> > > >>>>> public class TestReasonerProblem { >> > > >>>>> >> > > >>>>> @Test >> > > >>>>> public void test() throws Exception { >> > > >>>>> >> > > >>>>> Model model = ModelFactory.createDefaultModel(); >> > > >>>>> String file="clinical-factDCMapVent.owl"; >> > > >>>>> InputStream fis = >> > > >>>>>this.getClass().getResourceAsStream(file); >> > > >>>>> model.read(fis, null, "RDF/XML"); >> > > >>>>> OntModelSpec spec = OntModelSpec.OWL_MEM_RDFS_INF; >> > > >>>>> // OntModelSpec spec = OntModelSpec.OWL_DL_MEM_RDFS_INF; >> > > >>>>> // OntModelSpec spec = OntModelSpec.OWL_DL_MEM_TRANS_INF; >> > > >>>>> OntModel ontModel = ModelFactory.createOntologyModel(spec, >> > > >>>>>model); >> > > >>>>> >> > > >>>>> long ctms = System.currentTimeMillis(); >> > > >>>>> for (ExtendedIterator j = >> > > >>>>>ontModel.listHierarchyRootClasses(); j.hasNext(); ){ >> > > >>>>> j.next(); >> > > >>>>> } >> > > >>>>> System.out.printf("time: %d, totMem: %dk, usedMem: %dk\n", >> > > >>>>> (System.currentTimeMillis()-ctms), >> > > >>>>> (Runtime.getRuntime().totalMemory()/1024), >> > > >>>>> >> > > >>>>>((Runtime.getRuntime().totalMemory()-Runtime.getRuntime().freeMemory())/1024)); >> > > >>>>> } >> > > >>>>> >> > > >>>>> } >> > > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>>>> ------------------------------------ >> > > >>>>> >> > > >>>>> Yahoo! Groups Links >> > > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>> >> > > >>> >> > > >>> >> > > >>> ------------------------------------ >> > > >>> >> > > >>> Yahoo! Groups Links >> > > >>> >> > > >>> >> > > >>> >> > > > >> > > > >> > > > >> > > > >> > > > ------------------------------------ >> > > > >> > > > Yahoo! Groups Links >> > > > >> > > > >> > > > >> > > >> > >> > > > > > > >
