Claus Stadler created JENA-1848:
-----------------------------------
Summary: Trig Writer slow; doesn't scale to many graphs
Key: JENA-1848
URL: https://issues.apache.org/jira/browse/JENA-1848
Project: Apache Jena
Issue Type: Improvement
Components: RIOT
Affects Versions: Jena 3.14.0
Reporter: Claus Stadler
The following code for loading 1.000.000 graphs takes 1 minute on my notebook,
but I stopped my attempt of writing the data out as trig after several hours.
{code:java}
Dataset ds = RDFDataMgr.loadDataset("test-data.trig");
RDFDataMgr.write(new NullOutputStream(), ds, RDFFormat.TRIG_PRETTY);
{code}
In comparison, writing takes 2 seconds for me with RDFFormat.NQUADS.
The test data I used can be generated with this *gendata.sh* bash script:
{code:bash}
#!/bin/bash
MAX=${1:-10}
echo "@prefix eg: <http://www.example.org/> ."
for i in `seq 1 $MAX`; do
echo "<urn:graph-$i> { <urn:s-$i> eg:idx $i }"
done
{code}
Invoke the script the number of named graphs to generate, in my case I used
{code:bash}
./gendata.sh 1000000 > test-data.trig`
{code}
With the profiler I could trace the problem to code in *TurtleShell.java* which
repeatedly collects all one million graph names :
{code:java}
this.graphNames = (dsg != null) ? Iter.toSet(dsg.listGraphNodes()) : null ;`
{code}
https://github.com/apache/jena/blob/2a13a9c633f1c8661c1a446a9d98819391c09477/jena-arq/src/main/java/org/apache/jena/riot/writer/TurtleShell.java#L185
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
