On 29/08/2022 14:53, Simon Bin wrote:
I was asked to try it on my system (samsung 970 evo+ nvme, intel
11850h), but I used a slightly smaller data set (river_europe); it is
not quite as bad as on Lorenz' but the buffering would help
nevertheless:
main : river_europe-latest.osm.pbf.ttl.bz2 : 815.14 sec : 72,098,221
Triples : 88,449.21 per second : 0 errors : 10 warnings
fix/bzip2 : river_europe-latest.osm.pbf.ttl.bz2 : 376.64 sec : 72,098,221
Triples : 191,424.76 per second : 0 errors : 10 warnings
pbzip2 -dc river_europe-latest.osm.pbf.ttl.bz2 | : 155.24 sec : 72,098,221
Triples : 464,442.66 per second : 0 errors : 10 warnings
river_europe-latest.osm.pbf.ttl : 136.92 sec : 72,098,221
Triples : 526,587.26 per second : 0 errors : 10 warnings
Are these two datasets (this dataset and
river_planet-latest.osm.pbf.ttl) publicly availably?
Different datasets have different performance characteristics.
I'm not surprised BSBM is slower - it has a lot of large literals so
there is a lot of basic byte shifting.
I also tried on a laptop which typically have slower buses. (I had a
hardware crash a couple of weeks ago so I don't have the desktop I was
using for comparison but from memory, the 8yo desktop is faster for riot
parsing than the 1yo laptop.)
Andy
If you want excellent figure, use LUBM. It has a small node/triple ratio
(there are less bytes to shift) and high locality of URI use (better
memory cache usage). It is unrealistic for parsing and loading.
Cheers,
On Mon, 2022-08-29 at 13:09 +0200, Lorenz Buehmann wrote:
In addition I used the OS tool in a pipe:
bunzip2 -c river_planet-latest.osm.pbf.ttl.bz2 | riot --time --count
--syntax "Turtle"
Triples = 163,310,838
stdin : 717.78 sec : 163,310,838 Triples : 227,523.09 per
second : 0 errors : 31 warnings
unsurprisingly more or less exactly the time of decompression + the
parsing time of the uncompressed file - still way faster than the
Apache
Commons one, even with my suggested fix the OS variant is ~5min
faster
On 29.08.22 11:24, Lorenz Buehmann wrote:
riot --time --count river_planet-latest.osm.pbf.ttl
Triples = 163,310,838
351.00 sec : 163,310,838 Triples : 465,271.72 per second : 0 errors
:
31 warnings
riot --time --count river_planet-latest.osm.pbf.ttl.gz
Triples = 163,310,838
431.74 sec : 163,310,838 Triples : 378,258.50 per second : 0 errors
:
31 warnings
riot --time --count river_planet-latest.osm.pbf.ttl.bz2
Triples = 163,310,838
9,948.17 sec : 163,310,838 Triples : 16,416.17 per second : 0
errors :
31 warnings
Takes ages with Bzip2 ... there must be something going wrong ...
We checked code and the Apache Commons Compress docs, a colleague
spotted the hint at
https://commons.apache.org/proper/commons-compress/examples.html#Buffering
:
The stream classes all wrap around streams provided by the
calling
code and they work on them directly without any additional
buffering.
On the other hand most of them will benefit from buffering so it
is
highly recommended that users wrap their stream in
Buffered(In|Out)putStreams before using the Commons Compress API.
we were curious about this statement, checked
org.apache.jena.atlas.io.IO class and added one line in openFileEx
in = new BufferedInputStream(in);
which wraps the file stream before its passed to the decompressor
streams
Run again the parsing:
riot --time --count river_planet-latest.osm.pbf.ttl.bz2 (Jena
4.7.0-SNAPSHOT fork with a BufferedInputStream wrapping the file
stream in IO class)
Triples = 163,310,838
1,004.68 sec : 163,310,838 Triples : 162,550.10 per second : 0
errors
: 31 warnings
What do you think?
On 28.08.22 14:22, Andy Seaborne wrote:
If you are relying on Jena to do the bz2 decompress, then it is
using Commons Compress.
gz is done (via Commons Compress) in native code. I use gz and
if I
get a bz2 file, I decompress it with OS tools.
Could you try an experiment please?
Run on the same hardware as the loader was run:
riot --time --count river_planet-latest.osm.pbf.ttl
riot --time --count river_planet-latest.osm.pbf.ttl.bz2
Andy
gz vs plain: NVMe m2 SSD : Dell XPS 13 9310
riot --time --count .../BSBM/bsbm-25m.nt.gz
Triples = 24,997,044
118.02 sec : 24,997,044 Triples : 211,808.84 per second
riot --time --count .../BSBM/bsbm-25m.nt
Triples = 24,997,044
109.97 sec : 24,997,044 Triples : 227,314.05 per second
