________________________________
From: Dave Reynolds<[email protected]>
To: Milorad Tosic<[email protected]>
Cc: "[email protected]"<[email protected]>
Sent: Friday, December 16, 2011 4:41 PM
Subject: Re: comparison of JVM and Jena versions by memory usage
Hi,
On Fri, 2011-12-16 at 05:39 -0800, Milorad Tosic wrote:
Hi,
There is a short discussion on the former Jena Development/Support mailing list
related to memory usage of Jena OntModel using different versions of Jena. For
your information I have included a segment of the correspondence at the end of
the message. Since I had a problem that looks much like memory usage limit
overrun, I got interested in the problem.
I repeated the same experiment but with
different conclusions? I compared Jena 2.6.0 with Jena 2.6.4. I couldn't
notice the difference between Jena versions as reported in your
previous experiment (though I used the same code). Tried with several
variations of the OntModel creations styles but with minor differences
in memory usage only (within 10%). I run the experiment from within
Eclipse with JVM set to jre6 using compiler compliance level 1.5 (as
well as 1.6). The total memory usage was around 27975k.
Than, I changed the JVM to jdk 1.5. There was similar relative
difference between the Jena 2.6.0 and Jena 2.6.4 (within 10%). However,
the absolute amount of used memory dropped to about 3400k.
Than I again changed the JVM to jdk 1.6. As in all previous testing
configurations, there was similar relative difference between the Jena
2.6.0 and Jena 2.6.4 (within 10%). However, the absolute amount of used
memory was about 6501k.
Consequently, I have two problems now:
1) I couldn't repeat you experiment with the same results. What I am doing
wrong?
Assuming you are running the code that Rick originally posted then the
problem is that that uses a naive way to measure memory usage.
If I run with java 1.6 (sun-java on Linux) and the current Jena snapshot
and run in eclipse I get:
time: 19, totMem: 122688k, usedMem: 4859k
If I run identical code from the command line I get:
time: 12, totMem: 122688k, usedMem: 28548k
The difference is down to whether a garbage collect has kicked in. If I
force a System.gc() in that code then I get:
time: 86, totMem: 122688k, usedMem: 4006k
where the extra time comes from the garbage collect.
2) What should I do to put the memory usage under my control? Namely, 5
times larger memory usage cause my application to broke on machines
with limited memory usage.
I don't think there's any evidence that the problem in the thread that
you are referring to is still an issue.
You would need to explain more about what you are actually do in your
application before we could help with that. However, if you are
currently using inference and are having memory problems then avoiding
inference, or using a weaker inference setting, would be a good place to
start.
Dave
Thanks,
Milorad Tosic
--- In [email protected], Dave Reynolds<dave.e.reynolds@...> wrote:
Rick,
Thanks for letting us know, thanks for the clear report.
Dave
rspates2 wrote:
Dave,
> > I grabbed OWLProfile.java and reran my tests. It looks fixed
to me in terms of speed and memory use. We should be able to use 2.6.2
now.
Thanks for your swift resolution to the problem.
Rick
--- In [email protected], Dave Reynolds<dave.e.reynolds@> wrote:
Hi Rick,
A follow up on my earlier off-list response.
Thanks again for giving us a test case for this. It turns out not to be
a reasoner issue as such. There was a change in the internals of the
Ontology API which worked inefficiently when a reasoner is present. Ian
has checked in some patches for this which brings the memory use down a
lot and gives a 5x speed improvement.
It is not quite back to 2.6.0 timings but is a significant improvement.
We'll investigate a little more but this may be as close as we can get
for now. Let us know if this is enough to make it usable for you.
Cheers,
Dave
rspates2 wrote:
Dave,
The difference seems to be between 2.6.0 and 2.6.2,
OntModel.listClasses():
2.6.2 time: 2187, totMem: 19948k, usedMem: 13345k
2.6.0 time: 110, totMem: 5056k, usedMem: 3681k
2.5.7 time: 78, totMem: 5056k, usedMem: 3726k
listHierarchyRootClasses():
2.6.2 time: 2312, totMem: 20152k, usedMem: 13859k
2.6.0 time: 172, totMem: 5056k, usedMem: 4312k
2.5.7 time: 188, totMem: 5056k, usedMem: 4078k
Let me know if you need anything else.
Thanks,
Rick
--- In [email protected], Dave Reynolds<dave.e.reynolds@> wrote:
rspates2 wrote:
Hi,
> >>>>> We switched our application from Jena 2.5.7
to 2.6.2. For an operation on OntModel.listHierarchyRootClasses() with
an OntModelSpec of OWL_MEM_RDFS_INF, we saw a very significant increase
in the use of memory and processing time, including out-of-memory
condition for our full ontology. Using a subset of our ontology and the
code below to demonstrate, we measured processing times and memory use:
2.6.2 time: 2063, totMem: 20008k, usedMem: 14539k
2.5.7 time: 172, totMem: 5056k, usedMem: 3974k
Certainly a huge jump :(
> >>>>> The specifications OWL_MEM_RDFS_INF and
OWL_DL_MEM_RDFS_INF produced comparable results, while
OWL_DL_MEM_TRANS_INF at 2.6.2 was comparable to our 2.5.7 run with
OWL_MEM_RDFS_INF.
I can post the subset ontology if you need it (170
k), but thought there might be some known explanation you could provide
without running the demo code.
Any idea what is causing this?
No immediate ideas. The RFDS rule file itself hasn't changed in that
time frame. It is tricky to trace back code changes over that period
because we did a code reorg which hides a lot of the history. However,
from memory (and the release notes) I can't think of anything that
should have had an effect like that. There were changes around handling
validation of data types but that isn't relevant here. We generalized
handling of non-legal RDF triples but again that shouldn't be relevant.
Could you try two experiments?
First, would it be possible to time a simpler operation, such as listing
all classes, to eliminate listHierarchyRootClasses as a factor?
Secondly, is it possible to check with Jena 2.6.0 - that would separate
the Java 5 changes and the datatype validation from more recent tweaks?
If that's a problem then send me your ontology subset and I'll take a
look when I can.
Dave
Thanks for your help,
Rick
import java.io.InputStream;
import org.junit.Test;
import com.hp.hpl.jena.ontology.OntModel;
import com.hp.hpl.jena.ontology.OntModelSpec;
import com.hp.hpl.jena.rdf.model.Model;
import com.hp.hpl.jena.rdf.model.ModelFactory;
import com.hp.hpl.jena.util.iterator.ExtendedIterator;
public class TestReasonerProblem {
@Test
public void test() throws Exception {
Model model = ModelFactory.createDefaultModel();
String file="clinical-factDCMapVent.owl";
InputStream fis = this.getClass().getResourceAsStream(file);
model.read(fis, null, "RDF/XML");
OntModelSpec spec = OntModelSpec.OWL_MEM_RDFS_INF;
// OntModelSpec spec = OntModelSpec.OWL_DL_MEM_RDFS_INF;
// OntModelSpec spec = OntModelSpec.OWL_DL_MEM_TRANS_INF;
OntModel ontModel = ModelFactory.createOntologyModel(spec, model);
long ctms = System.currentTimeMillis();
for (ExtendedIterator j = ontModel.listHierarchyRootClasses();
j.hasNext(); ){
j.next();
}
System.out.printf("time: %d, totMem: %dk, usedMem: %dk\n",
(System.currentTimeMillis()-ctms),
(Runtime.getRuntime().totalMemory()/1024),
((Runtime.getRuntime().totalMemory()-Runtime.getRuntime().freeMemory())/1024));
}
}
------------------------------------
Yahoo! Groups Links
------------------------------------
Yahoo! Groups Links
------------------------------------
Yahoo! Groups Links
