[
https://issues.apache.org/jira/browse/JENA-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042544#comment-17042544
]
Andy Seaborne commented on JENA-1848:
-------------------------------------
[~Aklakan] - thank you for the complete report. I've reproduced it.
{{time riot --pretty trig test-data.trig >/dev/null}}
Computing the graph names set once, in the code further out works. It is
already doing that work in {{TriGWriter.write}}. This avoids the problem that
simply using {{dsg.containsGraph}} in
[TurtleShell.containedInOneGraph|https://github.com/apache/jena/blob/2a13a9c633f1c8661c1a446a9d98819391c09477/jena-arq/src/main/java/org/apache/jena/riot/writer/TurtleShell.java#L316]
which isn't necessarily fast for persistent databases.
General point: {{RDFFormat.TRIG_BLOCKS}} is a fast, scalable form of TriG. Not
as pretty, but pretty printing takes significant amounts of working space.
> Trig Writer slow; doesn't scale to many graphs
> ----------------------------------------------
>
> Key: JENA-1848
> URL: https://issues.apache.org/jira/browse/JENA-1848
> Project: Apache Jena
> Issue Type: Improvement
> Components: RIOT
> Affects Versions: Jena 3.14.0
> Reporter: Claus Stadler
> Priority: Major
>
> The following code for loading 1.000.000 graphs takes 1 minute on my
> notebook, but I stopped my attempt of writing the data out as trig after
> several hours.
> {code:java}
> Dataset ds = RDFDataMgr.loadDataset("test-data.trig");
> RDFDataMgr.write(new NullOutputStream(), ds, RDFFormat.TRIG_PRETTY);
> {code}
> In comparison, writing takes 2 seconds for me with RDFFormat.NQUADS.
> The test data I used can be generated with this *gendata.sh* bash script:
> {code:bash}
> #!/bin/bash
> MAX=${1:-10}
> echo "@prefix eg: <http://www.example.org/> ."
> for i in `seq 1 $MAX`; do
> echo "<urn:graph-$i> { <urn:s-$i> eg:idx $i }"
> done
> {code}
> Invoke the script with the number of named graphs to generate, in my case I
> used
> {code:bash}
> ./gendata.sh 1000000 > test-data.trig`
> {code}
> With the profiler I could trace the problem to code in
> [TurtleShell.java|https://github.com/apache/jena/blob/2a13a9c633f1c8661c1a446a9d98819391c09477/jena-arq/src/main/java/org/apache/jena/riot/writer/TurtleShell.java#L185]
> which repeatedly collects all one million graph names :
> {code:java}
> this.graphNames = (dsg != null) ? Iter.toSet(dsg.listGraphNodes()) : null ;`
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
