Hi, DELETE DATA solved my problem - the performance is very good, I am doing a test to delete in batches of 50 000 triples and it takes 20-30 second to delete one batch! My JVM has -Xmx30g, I will try to increase the batch size even more. Thank you Stephen!
2014-06-24 22:29 GMT+01:00 Stephen Allen <[email protected]>: > Also, use INSERT DATA as well for the insertion if you can. > > > On Tue, Jun 24, 2014 at 5:21 PM, Stephen Allen <[email protected]> wrote: > > > Hi Ewa, > > > > Try using the DELETE DATA query update form instead of DELETE WHERE. The > > DELETE DATA form is optimized for streaming and can handle many more > > triples than the WHERE form. The only limit on the number of triples to > be > > deleted is based on the amount of memory used in the heap by TDB's > > uncommitted blocks, and not based on any limit in the SPARQL parser. > > > > -Stephen > > > > > > On Tue, Jun 24, 2014 at 11:42 AM, Ewa Szwed <[email protected]> wrote: > > > >> Hi, > >> > >> More about my experiment case: > >> > >> My SPARQL delete file has 8000 triples: > >> > >> [root@freebase-m03 freebase_store_updates]# wc -l temp-file > >> 8002 temp-file > >> > >> temp-file Content sample: > >> > >> DELETE WHERE { > >> <http://rdf.freebase.com/ns/g.126tg6__p> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6__p> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6__r> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6__r> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6__r> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_fc> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_fc> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_fc> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_g0> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Autor"@es > . > >> <http://rdf.freebase.com/ns/g.126tg6_g0> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> > >> "Συγγραφέας"@el > >> . > >> <http://rdf.freebase.com/ns/g.126tg6_g3> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.0np9r> . > >> <http://rdf.freebase.com/ns/g.126tg6_g3> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Voice > >> actor"@en > >> . > >> <http://rdf.freebase.com/ns/g.126tg6_g3> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.0np9r> . > >> <http://rdf.freebase.com/ns/g.126tg6_gd> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_gd> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_gd> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_gj> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_gj> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_gj> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_js> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_js> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_js> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_kf> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_kf> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_kf> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_m6> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_m6> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_m6> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_m8> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_m8> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_m8> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mb> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mb> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_mb> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mf> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mf> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_mf> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_ml> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_ml> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_ml> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mn> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mn> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_mn> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mt> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mt> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> <http://rdf.freebase.com/ns/g.126tg6_mt> < > >> http://rdf.freebase.com/ns/common.notable_for.notable_object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mz> < > >> http://rdf.freebase.com/ns/common.notable_for.object> < > >> http://rdf.freebase.com/ns/m.09jd9nh> . > >> <http://rdf.freebase.com/ns/g.126tg6_mz> < > >> http://rdf.freebase.com/ns/common.notable_for.display_name> "Canonical > >> Version"@en . > >> > >> > >> Problematic command: > >> $TDB_UPDATE_DIR/tdbupdate --loc $FREEBASE_DATA_DIR > >> --update=$FS_UPDATER_HOME > >> /temp-file > >> > >> Error trace: > >> > >> 11:28:19 WARN TDB :: Transaction not commited or > >> aborted: > >> Transaction: 1 : Mode=WRITE : State=ACTIVE : > /data/servers/freebase_data/ > >> > >> Exception in thread "main" java.lang.StackOverflowError > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:195) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> at > >> > >> > org.apache.jena.atlas.iterator.RepeatApplyIterator.hasNext(RepeatApplyIterator.java:46) > >> at > >> > >> > com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable.hasNext(SolverLib.java:197) > >> > >> Software version: > >> more ReleaseNotes-Jena.txt > >> Release Notes > >> ============= > >> > >> ==== Jena 2.11.0 > >> > >> > >> Please note that when my temp-file was 5000 triples or smaller it worked > >> just fine. > >> > >> Please advise because I have a feeling I could increase the size to 10 > 000 > >> - 20 000 triples with performance improvement. > >> > >> When batch was: > >> > >> 500 triples, it took 20 seconds fro batch > >> > >> when > >> > >> 5000 triples it took 30 seconds so I expect good results for even bigger > >> batches but this StackOverFlow is a blocker. > >> > >> Best Regards, > >> > >> Ewa > >> > >> > >> 2014-06-24 14:33 GMT+01:00 Andy Seaborne <[email protected]>: > >> > >> > On 23/06/14 18:27, Ewa Szwed wrote: > >> > > >> >> Hi, > >> >> > >> >> I migrated my Freebase Jena TDB setup to SSD host and continued my > >> >> tdbupdate deletes experiments. The expectation was the transaction > >> times > >> >> are improved. I was able to see the improvements: > >> >> > >> >> batch 5000 triples - 1 min 20 sec / batch > >> >> > >> >> Now when I reconfigure my bath size to 8000 to see if the time stays > at > >> >> this level for single batch I see: > >> >> > >> >> > >> >> WARN TDB:: Transaction not commited or aborted: Transaction: 1 : > >> >> Mode=WRITE > >> >> : State=ACTIVE > >> >> : /data/servers/freebase_data/ > >> >> > >> >> Exception in thread "main" java.lang.StackOverflowError > >> >> at > >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> >> hasNext(RepeatApplyIterator.java:42) > >> >> at > >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. > >> >> hasNext(SolverLib.java:197) > >> >> at > >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> >> hasNext(RepeatApplyIterator.java:46) > >> >> at > >> >> com.hp.hpl.jena.tdb.solver.SolverLib$IterAbortable. > >> >> hasNext(SolverLib.java:197) > >> >> at > >> >> org.apache.jena.atlas.iterator.RepeatApplyIterator. > >> >> hasNext(RepeatApplyIterator.java:46) > >> >> > >> >> Is this a bug? > >> >> Please advise how should I proceed? > >> >> > >> > > >> > I'd need to know more about what's going to say but it isn't anything > I > >> > recognize. > >> > > >> > Could you provide the whole stack trace (well, the start and finish of > >> the > >> > boring bit :-) and details of the update being attempted. And could > you > >> > please confirm the version you're using? > >> > > >> > Thanks > >> > Andy > >> > > >> > > >> > > >> > > >> > > > > >
