dr0ptp4kt added a comment.

  The run with with buffer at 100000**0** and heap size at 31g and queue 
capacity at 4000 on the gaming-class desktop completed.
    Processing wikidump-000001332.ttl.gz
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
content="text&#47;html;charset=UTF-8"><title>blazegraph&trade; by SYSTAP</title
    ><body<p>totalElapsed=13580ms, elapsed=13580ms, connFlush=0ms, 
batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms</p
    ><hr><p>COMMIT: totalElapsed=266483ms, commitTime=1711304860167, 
    >Sun Mar 24 01:27:45 PM CDT 2024
    real    5690m30.371s
  ... which is 3.95 days. I'm trying again, but going back to the buffer 
capacity at 100000 instead of 100000**0** for one last comparison with these 
runs on this subset of munged data, and without any larger pause between 
batches of files (remember the previous run with buffer capacity at 100000 and 
31g heap and queue capacity at 4000 was done by first running files 1-150, then 
after coming back to the terminal sometime later resumed from file 151; but in 
the real world we usually hope to just let this thing run one file after 
another without any pause...in practice it could be that allowing the JVM time 
to heal itself created some artificial speed gains, but we'll see).
  Starting on Friday, March 29, 2024 at 1:40 PM CT...
    ubuntu22:~/rdf/dist/target/service-0.3.138-SNAPSHOT$ time ./loadData.sh -n 
wdq -d /mnt/firehose/munge_on_later_data_set -s 1 -e 1332
    Processing wikidump-000000001.ttl.gz
  I'll update when it's done. It should complete presumably sometime in the 
next 24 hours.



To: dr0ptp4kt
Cc: ssingh, bking, dr0ptp4kt, Aklapper, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, KimKelting, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

Reply via email to