On 27/07/16 21:16, Dick Murray wrote:
The MWE in the previous email will work with any even line text file and
will produce the odd [B values. I can't see anywhere obvious where the non
Jena code is creating them, just odd that there's so many of them!
Could be - it's just not a size that reminds me of anything.
OK, that knocks the DBB idea on the head!
I'll set the mapped symbol and play with batch sizes. Can the map location
be configured or will it go after the TDB location?
See BlockMgrJournal
(really, look at the code for answers - some of this hasn't been visited
for a while!)
It uses a file in the tmp director from java.io.tmpdir
Is "TDB2" what we discussed some time back? I'm happy to provide some
testing on that as I've ~2000 files to ETL via an automated process each
producing 3-4M quads...
Thanks for the offer. I need to find time to clean it up but I've used
it with Fuseki and loaded large files (sized up until I got bored rather
than some limit I hit). It is Apache licensed - it is not part of the
Jena project.
Andy
Thanks Dick.
On 27 Jul 2016 20:10, "Andy Seaborne" <a...@apache.org> wrote:
On 27/07/16 13:19, Dick Murray wrote:
;-) Yes I did. But then I switched to the actual files I need to import
and
they produce ~3.5M triples...
Using normal Jena 3.1 (i.e. no special context symbols set) the commit
after 100k triples works to import the file 10 times with the [B varying
between ~2Mb and ~4Mb. I'm currently testing a 20 instance pass.
A batched commit works for this bulk load because if it fails after a
batch
commit I can remove the graph.
For my understanding... TDB is holding the triples/block/journal in heap
until commit is called? But this doesn't account for the [B not being
cleared after a commit of 3.5M triples. It takes another pass plus ~2M
uncommited triples before I get an OOME.
And the [B have a strange average size. A block is 8K.
Digging around and there are some references made to the
DirectByteBuffers
causing issues. IBM
https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/excessive_native_memory_usage_by_directbytebuffers?lang=en
links the problem to;
Essentially the problem boils down to either:
1. There are too many DBBs being allocated (or they are too large),
and/or
2. The DBBs are not being cleared up quickly enough.
TDB does not use DirectByteBuffers unless you ask it to. They are not [B.
.hasArray is false.
.array() throws UnsupportedOperationException.
(Grep the code for "allocateDirect" and trace back the use of single use
of BufferAllocatorDirect to the journal in "direct" mode.)
I can believe that if activated, that the GC recycling would be slow. The
code ought to recycle them (beause yuo can't explicitly free them for some
weird reason - they little more than malloc).
But they are not being used unless you ask for them.
Journal entries are 8K unless they are commit records which are about 20
bytes (I think).
and recommends using -XX:MaxDirectMemorySize=1024m to poke the GC via
System.gc(). Not sure if GC1C helps because of it's new heap model...
Would it be possible to get Jena to write it's uncommitted triples to
disk
and then commit them to the TDB?
Set TDB.transactionJournalWriteBlockMode to "mapped". That uses a disk
file.
Ok it's slower than RAM but until they are
committed only one thread has visibility anyway? Could direct that at a
different disk as well...
Just before hitting send I'm at pass 13 and the [B maxed at just over 4Gb
before dropping back to 2Gb.
Or use TDB2 :-)
It has no problem loading 100m+ triples in a single transaction (the
space per transaction is fixed at about 80 bytes of transaction - disk
writes happen during the transaction not to a roll-forward journal). And it
should be a bit faster because writes happen once.
Just need to find time to clean it up ...
Andy
Dick.
On 27 July 2016 at 11:47, Andy Seaborne <a...@apache.org> wrote:
On 27/07/16 11:22, Dick Murray wrote:
Hello.
Something doesn't add up here... I've run repeated tests with the
following
MWE on a 16GB machine with -Xms8g -Xmx8g and the I always get an OOME.
What I don't understand is the size of [B increases with each pass
until
the OOME is thrown. The exact same process is run 5 times with a new
graph
for each set of triples.
There are ~3.5M triples added within the transaction from a file which
is
a
"simple" text based file (30Mb) which is read in line pairs.
Err - you said 200k quads earlier!
Set
TransactionManager.QueueBatchSize=0 ;
and break the load into small units for now and see if that helps.
One experiment would be to write the output to disk and load from a
program that only does the TDB part.
Andy
I've tested sequential loads of other text files (i.e. file x *5) and
other
text files loaded sequentally (i.e. file x, file y, file ...) and the
same
result is exhibited.
If I reduce -Xmx to 6g it will fail earlier.
Changing the GC using -XX:+UseGC1C doesn't alter the outcome.
I'm running on Ubuntu 16.04 with Java 1.8 and I can replicate this on
Centos 7 with Java 1.8.
Any ideas?
Regards Dick.
