Also, use INSERT DATA as well for the insertion if you can.
On Tue, Jun 24, 2014 at 5:21 PM, Stephen Allen <[email protected]> wrote: > Hi Ewa, > > Try using the DELETE DATA query update form instead of DELETE WHERE. The > DELETE DATA form is optimized for streaming and can handle many more > triples than the WHERE form. The only limit on the number of triples to be > deleted is based on the amount of memory used in the heap by TDB's > uncommitted blocks, and not based on any limit in the SPARQL parser. > > -Stephen > > > On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <[email protected]> wrote: > >> Hi, >> >> More about my experiment case: >> >> My SPARQL delete file has 8000 triples: >> >> [root@freebase-m03 freebase_store_updates]# wc -l temp-file >> 8002 temp-file >> >> temp-file Content sample: >> >> DELETE WHERE { >> <http://rdf.freebase.com/ns/g.126tg6__p> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6__p> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6__r> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6__r> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6__r> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_fc> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_fc> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_fc> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_g0> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es . >> <http://rdf.freebase.com/ns/g.126tg6_g0> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> >> "Συγγραφέας"@el >> . >> <http://rdf.freebase.com/ns/g.126tg6_g3> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.0np9r> . >> <http://rdf.freebase.com/ns/g.126tg6_g3> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice >> actor"@en >> . >> <http://rdf.freebase.com/ns/g.126tg6_g3> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.0np9r> . >> <http://rdf.freebase.com/ns/g.126tg6_gd> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_gd> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_gd> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_gj> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_gj> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_gj> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_js> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_js> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_js> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_kf> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_kf> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_kf> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_m6> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_m6> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_m6> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_m8> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_m8> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_m8> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mb> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mb> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_mb> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mf> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mf> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_mf> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_ml> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_ml> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_ml> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mn> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mn> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_mn> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mt> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mt> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> <http://rdf.freebase.com/ns/g.126tg6_mt> < >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mz> < >> http://rdf.freebase.com/ns/common.notable_for.object> < >> http://rdf.freebase.com/ns/m.09jd9nh> . >> <http://rdf.freebase.com/ns/g.126tg6_mz> < >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical >> Version"@en . >> >> >> Problematic command: >> $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR >> --update=$FS_UPDATER_HOME >> /temp-file >> >> Error trace: >> >> 11:28:19 WARN TDB :: Transaction not commited or >> aborted: >> Transaction: 1 : Mode=WRITE : State=ACTIVE : /data/servers/freebase_data/ >> >> Exception in thread "main" java.lang.StackOverflowError >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) >> >> Software version: >> more ReleaseNotes-Jena.txt >> Release Notes >> ============= >> >> ==== Jena 2.11.0 >> >> >> Please note that when my temp-file was 5000 triples or smaller it worked >> just fine. >> >> Please advise because I have a feeling I could increase the size to 10 000 >> - 20 000 triples with performance improvement. >> >> When batch was: >> >> 500 triples, it took 20 seconds fro batch >> >> when >> >> 5000 triples it took 30 seconds so I expect good results for even bigger >> batches but this StackOverFlow is a blocker. >> >> Best Regards, >> >> Ewa >> >> >> 2014-06-24 14:33 GMT+01:00 Andy Seaborne <[email protected]>: >> >> > On 23/06/14 18:27, Ewa Szwed wrote: >> > >> >> Hi, >> >> >> >> I migrated my Freebase Jena TDB setup to SSD host and continued my >> >> tdbupdate deletes experiments. The expectation was the transaction >> times >> >> are improved. I was able to see the improvements: >> >> >> >> batch 5000 triples - 1 min 20 sec / batch >> >> >> >> Now when I reconfigure my bath size to 8000 to see if the time stays at >> >> this level for single batch I see: >> >> >> >> >> >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 : >> >> Mode=WRITE >> >> : State=ACTIVE >> >> : /data/servers/freebase_data/ >> >> >> >> Exception in thread "main" java.lang.StackOverflowError >> >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. >> >> hasNext(RepeatApplyIterator.java:42) >> >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. >> >> hasNext(SolverLib.java:197) >> >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. >> >> hasNext(RepeatApplyIterator.java:46) >> >> at >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. >> >> hasNext(SolverLib.java:197) >> >> at >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. >> >> hasNext(RepeatApplyIterator.java:46) >> >> >> >> Is this a bug? >> >> Please advise how should I proceed? >> >> >> > >> > I'd need to know more about what's going to say but it isn't anything I >> > recognize. >> > >> > Could you provide the whole stack trace (well, the start and finish of >> the >> > boring bit :-) and details of the update being attempted. And could you >> > please confirm the version you're using? >> > >> > Thanks >> > Andy >> > >> > >> > >> > >> > >
