[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt closed this task as "Resolved". dr0ptp4kt claimed this task. dr0ptp4kt added a comment. Thanks @RKemper ! These speed gains are welcome news. We should discuss in a near future meeting if there are any further actions. I can see how we may want to set the bufferCapacity to

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a comment. Mirroring comment in T359062#9783010 : > And for the second run in T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors) we saw

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a comment. In T362920#9776418 , @RKemper wrote: > @dr0ptp4kt > >> we saw that this took about 3702 minutes, or about 2.57 //hours// > > Typo you'll want to fix here and in the original: 2.57 **days** I think

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

RKemper added a comment. Kicked off the second run like so: (1) `vi /etc/wdqs/RWStore.wikidata.properties` (added a trailing 0 to buffersize) (2) `date | tee /home/ryankemper/loadData2.log; time sudo /home/ryankemper/loadData.sh -n wdq -d /srv/T362920/nt_wd_schol -s 0 -e 0 2>&1 |

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

RKemper added a comment. @dr0ptp4kt > we saw that this took about 3702 minutes, or about 2.57 //hours// Typo you'll want to fix here and in the original: 2.57 **days** TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a comment. Mirroring comment in T359062#9775908 : > In T362920 : Benchmark Blazegraph import with increased buffer capacity (and other factors) we saw that this took about 3702

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE,

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

gerritbot added a comment. Change #1026668 **abandoned** by Ryan Kemper: [operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021 Reason: don't need nfs for graph split reload https://gerrit.wikimedia.org/r/1026668 TASK DETAIL

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:48Z] END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on wdqs2023.codfw.wmnet with reason: T362920 TASK DETAIL

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:44Z] START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on wdqs2023.codfw.wmnet with reason: T362920 TASK DETAIL

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

gerritbot added a comment. Change #1027001 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: switch wdqs2023 to graph split host https://gerrit.wikimedia.org/r/1027001 TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:27:31Z] T362920 [wdqs] Depooled `wdqs2023` in preparation to switch it to a graph split host TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

gerritbot added a comment. Change #1027001 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wdqs: switch wdqs2023 to graph split host https://gerrit.wikimedia.org/r/1027001 TASK DETAIL https://phabricator.wikimedia.org/T362920

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a comment. Another thing that can be nice for figuring out stuff later is to add some timing and a simple log file. A command like the following was helpful when I was trying this out on the gaming-class desktop (you may not need this if your tmux session lets you scroll

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a comment. @RKemper I think that's captured in P54284 . If you need to get a copy of the files, there's a pointer in T350106#9381611 for how one might go about copying from HDFS

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

RKemper added a subscriber: dcausse. RKemper added a comment. I'm realizing I don't remember enough about how we load specific graph splits (scholarly vs main). But it's possible we won't need the above nfs patch if our previous process was to manually download the relevant dump file.

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, Isabelladantes1983,

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

gerritbot added a comment. Change #1026668 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021 https://gerrit.wikimedia.org/r/1026668 TASK DETAIL https://phabricator.wikimedia.org/T362920

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

RKemper added a comment. Actually we'll use `wdqs1021` since we're not confident NFS will work seamlessly between eqiad and codfw (we use nsf to procure the dumps that the data-reload is run from). Gearing up to kick off a data reload on `wdqs1021` after a quick patch to enable NFS on

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

RKemper added a comment. We'll use wdqs2023 to compare against wdqs1023 (same hardware). 1023 is scholarly-articles so that's what we'll want to load 2023 with. TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

Gehel triaged this task as "Low" priority. TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen,

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dr0ptp4kt Cc: Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

dr0ptp4kt renamed this task from "Benchmark Blazegraph import with increased buffer capacity" to "Benchmark Blazegraph import with increased buffer capacity (and other factors)". TASK DETAIL https://phabricator.wikimedia.org/T362920 EMAIL PREFERENCES