On 29/03/16 18:19, Adrian Gschwend wrote:
On 29.03.16 19:13, Andy Seaborne wrote:
Hi Andy,
So JSON-LD does not stream end-to-end.
Ok I thought something like this. We have the same problem with the
JavaScript JSON-LD library.
At 800Mb I would have expected a large enough heap to work for N-triples
output. Is the file available online anywhere?
the generated file is here:
http://www.eisenhutinformatik.ch/tmp/swissNAMES3D_LV03.zip
(and is the JSON-LD one big object? It is not really JSON sweet spot
for large objects)
I'm not really into JSON-LD details so not sure
The start of the file is somewhat long literal heavy for the geo data.
There are some serious
I trace the paring and it dives into
com.fasterxml.jackson.core.JsonParser and starts to build a java object
for the JSON input.
So even a large heap does not seem to get the JSON in and processing
from parsed JSON through JSON-LD to RDF never happens. I fed it through
YourKit to profile and the java heap is getting hit hard. My guess is
that the java data structure is not particular compact.
It does seem to get into some kind of GC hell as the oldest GC
generation grows which I think is the cause of lost of CPU cycles and
little real progress.
Andy
regards
Adrian
