Hi Andy, thanks for this.
I found a way to use the TDBLoader programmatically:
https://github.com/marco-brandizi/rdfutils/blob/master/rdfutils-jena/src/main/java/info/marcobrandizi/rdfutils/jena/elt/TDBLoadingHandler.java#L44
I use this class in a way similar to this:
https://github.com/marco-brandizi/rdfutils/blob/master/rdfutils-jena/src/test/java/info/marcobrandizi/rdfutils/jena/elt/RDFImporterTest.java#L95
https://github.com/marco-brandizi/rdfutils/blob/master/rdfutils-jena/src/main/java/info/marcobrandizi/rdfutils/jena/elt/RDFImporter.java
More in detail, I have an application that creates Jena's (default)
Model instances from converting non-RDF data, then it sends such models
to the TDBLoadingHandler above for saving data into a common TDB. As you
can see, now I've serialised the writings and I'm fine with it
(multi-threaded writing doesn't seem supported in TDB).
Following your comments, I've also tried both the version above in
TDBLoadingHandler, which uses the bulk loader, and the one that uses the
transactional approach (the commented code in the TDBLoadingHandler).
Initially I was sending Model instances with a size of 10-30k triples
and, in that case, the transactional way seems to be quite slower, now
I'm working with 500k triple chunks and indeed this way the speeds in
the two cases are pretty similar, but I get memory overflow errors when
I try to use the transactional version (while the bulk loading works
fine). So, I think I will keep the bulk loader method for the time
being, at least if my code above is correct enough.
Marco.
On 03/01/2018 10:20, Andy Seaborne wrote:
Marco,
There are some things that are no clear:
* 2G of Turtle - how many triples?
* where does the multithreading come into it?
* what's 'model'?
* what's the setup? hardware? jena version? java heap size?
Bulk loading is faster on an empty database, otherwise it isn't doing
anything special. Transactions are safer.
One single write transaction for all the data to be added would be the
right way to do it. A transaction has some overhead so very small
transactions incur costs but with a reasonable size, x10 is not likely.
Andy
On 02/01/18 22:13, Zak Mc Kracken wrote:
Hi all again,
I'm trying to ask again about the problem below. It surprises me that
is so slow (like 2k triples/s, when the tdbloader tool is 10x faster).
I've tried to use org.apache.jena.tdb.TDBLoader, but I haven't clear
how to obtain a DatasetGraphTDB or a GraphNonTxnTDB from the path of
a TDB location.
Thanks in advance for any help,
Marco
On 23/12/2017 13:09, Zak Mc Kracken wrote:
Hi all,
I've an application where exporting threads are producing Model
instances of a pre-configured size, then I want to write those
models into a TDB.
For the moment, I'm using this (I believe, rather canonical) code
for the writing (dataSet is shared between threads:
this.dataSet.begin ( ReadWrite.WRITE );
try {
Model dmodel = this.dataSet.getDefaultModel ();
dmodel.add ( model );
this.dataSet.commit ();
}
finally {
this.dataSet.end ();
}
It's extremely slow. When tested with about 2G of Turtle data, it is
still running after hours. The same data exported to a .ttl and then
loaded with tdbloader take a couple of minutes. Am I doing something
wrong? Is the transactional approach inherently slower? Should I
call the TDBLoader instead (the one used by the command line tool)?
Note that I'm OK to force everything to one thread only (or to
serialise a couple of threads, as the code above seem to enforce),
the application is going to be used in different export use cases
and in some of them it will be truly parallel (eg, saving data on
different files).
Thanks in advance for any help.
Marco.
