[
https://issues.apache.org/jira/browse/JENA-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Seaborne resolved JENA-1848.
---------------------------------
Fix Version/s: Jena 3.15.0
Assignee: Andy Seaborne
Resolution: Fixed
> Trig Writer slow; doesn't scale to many graphs
> ----------------------------------------------
>
> Key: JENA-1848
> URL: https://issues.apache.org/jira/browse/JENA-1848
> Project: Apache Jena
> Issue Type: Improvement
> Components: RIOT
> Affects Versions: Jena 3.14.0
> Reporter: Claus Stadler
> Assignee: Andy Seaborne
> Priority: Major
> Fix For: Jena 3.15.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The following code for loading 1.000.000 graphs takes 1 minute on my
> notebook, but I stopped my attempt of writing the data out as trig after
> several hours.
> {code:java}
> Dataset ds = RDFDataMgr.loadDataset("test-data.trig");
> RDFDataMgr.write(new NullOutputStream(), ds, RDFFormat.TRIG_PRETTY);
> {code}
> In comparison, writing takes 2 seconds for me with RDFFormat.NQUADS.
> The test data I used can be generated with this *gendata.sh* bash script:
> {code:bash}
> #!/bin/bash
> MAX=${1:-10}
> echo "@prefix eg: <http://www.example.org/> ."
> for i in `seq 1 $MAX`; do
> echo "<urn:graph-$i> { <urn:s-$i> eg:idx $i }"
> done
> {code}
> Invoke the script with the number of named graphs to generate, in my case I
> used
> {code:bash}
> ./gendata.sh 1000000 > test-data.trig`
> {code}
> With the profiler I could trace the problem to code in
> [TurtleShell.java|https://github.com/apache/jena/blob/2a13a9c633f1c8661c1a446a9d98819391c09477/jena-arq/src/main/java/org/apache/jena/riot/writer/TurtleShell.java#L185]
> which repeatedly collects all one million graph names :
> {code:java}
> this.graphNames = (dsg != null) ? Iter.toSet(dsg.listGraphNodes()) : null ;`
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
