Hi Ewa, Try using the DELETE DATA query update form instead of DELETE WHERE. The DELETE DATA form is optimized for streaming and can handle many more triples than the WHERE form. The only limit on the number of triples to be deleted is based on the amount of memory used in the heap by TDB's uncommitted blocks, and not based on any limit in the SPARQL parser.
-Stephen On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <[email protected]> wrote: > Hi, > > More about my experiment case: > > My SPARQL delete file has 8000 triples: > > [root@freebase-m03 freebase_store_updates]# wc -l temp-file > 8002 temp-file > > temp-file Content sample: > > DELETE WHERE { > <http://rdf.freebase.com/ns/g.126tg6__p> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6__p> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6__r> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6__r> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6__r> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_fc> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_fc> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_fc> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_g0> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es . > <http://rdf.freebase.com/ns/g.126tg6_g0> < > http://rdf.freebase.com/ns/common.notable_for.display_name> > "Συγγραφέας"@el > . > <http://rdf.freebase.com/ns/g.126tg6_g3> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.0np9r> . > <http://rdf.freebase.com/ns/g.126tg6_g3> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice > actor"@en > . > <http://rdf.freebase.com/ns/g.126tg6_g3> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.0np9r> . > <http://rdf.freebase.com/ns/g.126tg6_gd> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_gd> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_gd> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_gj> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_gj> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_gj> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_js> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_js> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_js> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_kf> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_kf> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_kf> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_m6> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_m6> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_m6> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_m8> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_m8> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_m8> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mb> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mb> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_mb> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mf> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mf> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_mf> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_ml> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_ml> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_ml> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mn> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mn> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_mn> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mt> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mt> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > <http://rdf.freebase.com/ns/g.126tg6_mt> < > http://rdf.freebase.com/ns/common.notable_for.notable_object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mz> < > http://rdf.freebase.com/ns/common.notable_for.object> < > http://rdf.freebase.com/ns/m.09jd9nh> . > <http://rdf.freebase.com/ns/g.126tg6_mz> < > http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > Version"@en . > > > Problematic command: > $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR > --update=$FS_UPDATER_HOME > /temp-file > > Error trace: > > 11:28:19 WARN TDB :: Transaction not commited or aborted: > Transaction: 1 : Mode=WRITE : State=ACTIVE : /data/servers/freebase_data/ > > Exception in thread "main" java.lang.StackOverflowError > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > at > > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > at > > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > > Software version: > more ReleaseNotes-Jena.txt > Release Notes > ============= > > ==== Jena 2.11.0 > > > Please note that when my temp-file was 5000 triples or smaller it worked > just fine. > > Please advise because I have a feeling I could increase the size to 10 000 > - 20 000 triples with performance improvement. > > When batch was: > > 500 triples, it took 20 seconds fro batch > > when > > 5000 triples it took 30 seconds so I expect good results for even bigger > batches but this StackOverFlow is a blocker. > > Best Regards, > > Ewa > > > 2014-06-24 14:33 GMT+01:00 Andy Seaborne <[email protected]>: > > > On 23/06/14 18:27, Ewa Szwed wrote: > > > >> Hi, > >> > >> I migrated my Freebase Jena TDB setup to SSD host and continued my > >> tdbupdate deletes experiments. The expectation was the transaction times > >> are improved. I was able to see the improvements: > >> > >> batch 5000 triples - 1 min 20 sec / batch > >> > >> Now when I reconfigure my bath size to 8000 to see if the time stays at > >> this level for single batch I see: > >> > >> > >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 : > >> Mode=WRITE > >> : State=ACTIVE > >> : /data/servers/freebase_data/ > >> > >> Exception in thread "main" java.lang.StackOverflowError > >> at > >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> hasNext(RepeatApplyIterator.java:42) > >> at > >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. > >> hasNext(SolverLib.java:197) > >> at > >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> hasNext(RepeatApplyIterator.java:46) > >> at > >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. > >> hasNext(SolverLib.java:197) > >> at > >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> hasNext(RepeatApplyIterator.java:46) > >> > >> Is this a bug? > >> Please advise how should I proceed? > >> > > > > I'd need to know more about what's going to say but it isn't anything I > > recognize. > > > > Could you provide the whole stack trace (well, the start and finish of > the > > boring bit :-) and details of the update being attempted. And could you > > please confirm the version you're using? > > > > Thanks > > Andy > > > > > > > > >
