Hi Adrian, Do you have an example of such an update?
Some cases can't stream, but it is possible some cases aren't streaming when they could.
Or the whole transaction is quite large which is where TDB2 comes in. On 02/03/18 09:21, Adrian Gschwend wrote:
Hi group, I had some discussions with Andy before outside the list. I have a pipeline that creates a bunch of new triples via CONSTRUCT or INSERT queries. The patterns are in my opinion pretty easy to stream, if I do not do an insert but just a count, the query executes fine in TDB. As soon as I do constructs on top of that it runs forever, takes CPU like crazy and eventually runs out of memory.
It sounds like the heap size isn't big enough. Sometimes its just a matter of increasing by a small amount. Java exhibits this when the heap is nearly full.
Now my guess is that it tries to create everything in-memory which is probably simply too much. From some googling I found: https://issues.apache.org/jira/browse/JENA-329 https://issues.apache.org/jira/browse/JENA-205 if I get that correctly there is streaming for this kind of queries but I can't figure out if I can use that in tdb cli interfaces like tdbupdate. Is there a way to use that from cli? My pipelines are run in batches so I don't really care about speed, I just care that the queries finish.
regards Adrian
