Re: Jena TDB OOME GC overhead limit exceeded

On 27/07/16 21:16, Dick Murray wrote: The MWE in the previous email will work with any even line text file and will produce the odd [B values. I can't see anywhere obvious where the non Jena code is creating them, just odd that there's so many of them! Could be - it's just not a size that

Re: Jena TDB OOME GC overhead limit exceeded

The MWE in the previous email will work with any even line text file and will produce the odd [B values. I can't see anywhere obvious where the non Jena code is creating them, just odd that there's so many of them! OK, that knocks the DBB idea on the head! I'll set the mapped symbol and play

Re: Jena TDB OOME GC overhead limit exceeded

On 27/07/16 13:19, Dick Murray wrote: ;-) Yes I did. But then I switched to the actual files I need to import and they produce ~3.5M triples... Using normal Jena 3.1 (i.e. no special context symbols set) the commit after 100k triples works to import the file 10 times with the [B varying between

Re: Jena TDB OOME GC overhead limit exceeded

;-) Yes I did. But then I switched to the actual files I need to import and they produce ~3.5M triples... Using normal Jena 3.1 (i.e. no special context symbols set) the commit after 100k triples works to import the file 10 times with the [B varying between ~2Mb and ~4Mb. I'm currently testing a

Re: Jena TDB OOME GC overhead limit exceeded

On 27/07/16 11:22, Dick Murray wrote: Hello. Something doesn't add up here... I've run repeated tests with the following MWE on a 16GB machine with -Xms8g -Xmx8g and the I always get an OOME. What I don't understand is the size of [B increases with each pass until the OOME is thrown. The exact

Re: Jena TDB OOME GC overhead limit exceeded

Hello. Something doesn't add up here... I've run repeated tests with the following MWE on a 16GB machine with -Xms8g -Xmx8g and the I always get an OOME. What I don't understand is the size of [B increases with each pass until the OOME is thrown. The exact same process is run 5 times with a new

Re: Jena TDB OOME GC overhead limit exceeded

On 26/07/16 16:51, Dick Murray wrote: Ok, I set that option and I get different OOME from the direct buffer memory. You now have: > java.lang.OutOfMemoryError: > Direct buffer memory So that means the direct memory space has run out, not the heap. You can increase direct memory but that

Re: Jena TDB OOME GC overhead limit exceeded

Ok, I set that option and I get different OOME from the direct buffer memory. I then changed the GC using -XX:+UseG1GC (which still throws the OOME) and I don't get why it's throwing the OOME.!? [Eden: 2372.0M(2372.0M)->0.0B(1036.0M) Survivors: 84.0M->196.0M Heap:

Re: Jena TDB OOME GC overhead limit exceeded

To build clean locally do the following: at the top level: not jena-tdb mvn clean install -Pbootstrap mvn install -Pdev (or "mvn clean install" but that builds and tests a lot more) Andy On 26/07/16 14:25, Andy Seaborne wrote: On 26/07/16 14:11, Dick Murray wrote: Hi. I'll set

Re: Jena TDB OOME GC overhead limit exceeded

On 26/07/16 14:11, Dick Murray wrote: Hi. I'll set that and run the process again. As an aside I just pulled Master and TDB won't compile because it's can't find MultiSet? Are there notes on getting the Jena GIT into Eclipse? I want to put a count on the BufferAllocatorMem to see what it's

Re: Jena TDB OOME GC overhead limit exceeded

On 26/07/16 10:51, Dick Murray wrote: Hi. Where do you set "transactionJournalWriteBlockMode" please? We don't - its off by default. It's a symbol you can set in the global context. TDB.transactionJournalWriteBlockMode It is the only place that triggers DirectByteBuffers in TDB which I

Re: Jena TDB OOME GC overhead limit exceeded

Hi. Where do you set "transactionJournalWriteBlockMode" please? Would you expect to see a large number of [B heap entries in a 3.1 TDB? Dick. On 26 July 2016 at 10:39, Andy Seaborne wrote: > Dick, > > The report is embedded in your application setup with a lot of >

Re: Jena TDB OOME GC overhead limit exceeded

Dick, The report is embedded in your application setup with a lot of "org.iungo.dataset.bulkload" Just because the OOME occurs in TDB does not mean that the space is consumed in TDB - there may be a bigger memory hog elsewhere. Could you produce an RDF example? Maybe that file, already

Jena TDB OOME GC overhead limit exceeded

Hi. I've got a repeatable problem with Jena 3.1 when performing a bulk load. The bulk load converts a file into ~200k quads and adds them to a TDB instance within a normal begin write, add quads and commit. Initially this completes in 30-40 seconds, However if I repeat the process (with the same