I can't see how the tdb
caching itself could led to a OOM on all tests after a second or the like:
Enrico - sorry, nor can I.
On 64 bit, as you have, the node table cache is still in-heap even if
indexes are not, but if changing the heap size does not make a
difference, then it does not look like accumulated TDB junk.
If all the tests fail and fail at the same time, in the same place,
regardless of heap, it does suggest that the test itself is causing
massive allocation somehow.
As it seems to fail quickly, can you point jvisualvm/YourKit/... at a
test run? (If you find it is TDB contributing, then do let me know.)
Andy
On 06/12/13 21:45, Enrico Daga wrote:
Hi Andy, all,
On 6 December 2013 09:28, Andy Seaborne <[email protected]> wrote:
On 05/12/13 22:56, Enrico Daga wrote:
Hi all,
I am having difficulties compiling clerezza, precisely
[INFO] Building Clerezza - SCB Jena TDB Storage Provider 0.7-SNAPSHOT
...
TdbTcProviderTest>TcProviderTest.testGraphDeletion:433 ? OutOfMemory
Java
heap...
TdbTcProviderTest>TcProviderTest.testGetGraph:99 ? OutOfMemory Java
heap
space
TdbTcProviderTest>TcProviderTest.testGetTriplesGraph:459 ? OutOfMemory
Java he...
TdbTcProviderTest>TcProviderTest.testGraphIsNotMutable:339 ?
OutOfMemory
Java ...
TdbTcProviderTest.testCreateGraph ? OutOfMemory Java heap space
TdbTcProviderTest.testGetTriplesMGraph ? OutOfMemory Java heap space
TdbTcProviderTest.testGetTriples ? OutOfMemory Java heap space
TdbTcProviderTest.testDeleteEntity ? OutOfMemory Java heap space
TdbTcProviderTest.testCreateMGraphNoDuplicateNames ? OutOfMemory Java
heap spa...
TdbTcProviderTest.testCreateMGraphExtended ? OutOfMemory Java heap
space
TdbTcProviderTest.testGetMGraph ? OutOfMemory Java heap space
TdbTcProviderTest.testCreateGraphNoDuplicateNames ? OutOfMemory Java
heap
spac...
TdbTcProviderTest.testCreateGraphWithInitialCollection ? OutOfMemory
Java
heap...
TdbTcProviderTest.testCreateGraphExtended ? OutOfMemory Java heap
space
I tried increasing the heap until 3g (I have 4 on this machine).
Is your machine a 32bit machine with 32 bit java? I'm unclear here
because you say heap is 3G and I thought the 32 bit limit was 1.5G-ish.
$ mvn --version && java -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
13:51:28+0000)
Maven home: /opt/local/share/java/maven3
Java version: 1.6.0_65, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x", version: "10.9", arch: "x86_64", family: "mac"
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
Does change in heap size move the point of failure?
I checked with 1g and 3g and no, changing the heap size does not change
anything, all tests fail (after the same time each)
(TDB can't use memory mapped files on a 32 bit system so it uses old-style
file IO and caches in heap; downside, uses heap.)
Otherwise - internally there is a cache of open databases because opening
one, and having it filesystem-cache-cold is prohibitively expensive.
If the tests are creating different DB each time, this might be a partial
cause of heap usage.
This is the case, at a first look [1].
There is StoreConnection.reset() and related operations to carefully
manipulate this cache. TDB uses these for @BeforeClass/@AfterClass clean
tests
For testing, in-memory TDB databases are very useful. The default is a
fresh, uncached daatset on each call of TDBFactory.createDataset().
Your disk will rattle less. The tests will run faster. They have been
reported as faster than normal Jena in-memory datasets but I don't see why.
They work by having a RAM disk like are that bytes are written into and
copied out of so it is good simulation of a disk - no accidental sharing of
data.
I understand, having tests with an in memory TDB would be better. But there
should be something wrong with the tests, because I can't see how the tdb
caching itself could led to a OOM on all tests after a second or the like:
testCreateGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 1.125 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetTriplesMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 0.948 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetTriples(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 1.108 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testDeleteEntity(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 0.95 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateMGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 1.101 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateMGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 0.927 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testGetMGraph(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 1.088 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphNoDuplicateNames(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 0.988 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphWithInitialCollection(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 1.256 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
testCreateGraphExtended(org.apache.clerezza.rdf.jena.tdb.storage.TdbTcProviderTest)
Time elapsed: 0.935 sec <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
Bests,
Enrico
Andy
Any hint?
Thank you!
Enrico
[1]
https://git-wip-us.apache.org/repos/svn?p=clerezza.git;a=blob;f=parent/rdf.jena.tdb.storage/src/test/java/org/apache/clerezza/rdf/jena/tdb/storage/TdbTcProviderTest.java;h=3ab68476ec397f63715155c51e9c08c9e773681a;hb=HEAD
