[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-09 Thread dr0ptp4kt 
dr0ptp4kt closed this task as "Resolved".
dr0ptp4kt claimed this task.
dr0ptp4kt added a comment.


  Thanks @RKemper ! These speed gains are welcome news. We should discuss in a 
near future meeting if there are any further actions. I can see how we may want 
to set the bufferCapacity to 10**0** for imports, whereas we may want to 
just continue running with a bufferCapacity of 10 once a node is in serving 
mode, but good topic for discussion.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-09 Thread dr0ptp4kt 
dr0ptp4kt added a comment.


  Mirroring comment in T359062#9783010 
:
  
  > And for the second run in T362920: Benchmark Blazegraph import with 
increased buffer capacity (and other factors) 
 we saw that this took about 3089 
minutes, or about 2.**15** days, for the scholarly article entity graph with 
the CPU governor change (described in T336443#9726600 
 ) plus the bufferCapacity 
at 10**0** on wdqs2023.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-07 Thread dr0ptp4kt 
dr0ptp4kt added a comment.


  In T362920#9776418 , 
@RKemper wrote:
  
  > @dr0ptp4kt
  >
  >> we saw that this took about 3702 minutes, or about 2.57 //hours//
  >
  > Typo you'll want to fix here and in the original: 2.57 **days**
  
  I think this is what is referred to as wishful thinking! Okay, updated the 
comment in the other ticket's comment and in the comment up above.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-07 Thread RKemper 
RKemper added a comment.


  Kicked off the second run like so:
  
  (1) `vi /etc/wdqs/RWStore.wikidata.properties` (added a trailing 0 to 
buffersize)
  
  (2) `date | tee /home/ryankemper/loadData2.log; time sudo 
/home/ryankemper/loadData.sh -n wdq -d /srv/T362920/nt_wd_schol -s 0 -e 0 2>&1 
| tee -a /home/ryankemper/loadData2.log; time sudo /home/ryankemper/loadData.sh 
-n wdq -d /srv/T362920/nt_wd_schol 2>&1 | tee -a 
/home/ryankemper/loadData2.log; echo -en "\nrun finished at " | tee 
/home/ryankemper/loadData2.log; date | tee /home/ryankemper/loadData2.log`

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-07 Thread RKemper 
RKemper added a comment.


  @dr0ptp4kt
  
  > we saw that this took about 3702 minutes, or about 2.57 //hours//
  
  Typo you'll want to fix here and in the original: 2.57 **days**

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-06 Thread dr0ptp4kt 
dr0ptp4kt added a comment.


  Mirroring comment in T359062#9775908 
:
  
  > In T362920 : Benchmark 
Blazegraph import with increased buffer capacity (and other factors) we saw 
that this took about 3702 minutes, or about 2.57 hours, for the scholarly 
article entity with the CPU governor change (described in T336443#9726600 
 ) alone on wdqs2023.
  
  The count matches T359062#9695544 
.
  
select (count(*) as ?ct)
where {?s ?p ?o}

7643858078

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread Maintenance_bot 
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
S8321414, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 
Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 
786, Biggs657, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan, Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread gerritbot 
gerritbot added a comment.


  Change #1026668 **abandoned** by Ryan Kemper:
  
  [operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021
  
  Reason:
  
  don't need nfs for graph split reload
  
  https://gerrit.wikimedia.org/r/1026668

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:48Z] 
 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) 
for 6 days, 0:00:00 on wdqs2023.codfw.wmnet with reason: T362920 


TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:38:44Z] 
 START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 
on wdqs2023.codfw.wmnet with reason: T362920 


TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread gerritbot 
gerritbot added a comment.


  Change #1027001 **merged** by Ryan Kemper:
  
  [operations/puppet@production] wdqs: switch wdqs2023 to graph split host
  
  https://gerrit.wikimedia.org/r/1027001

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2024-05-03T21:27:31Z]  
T362920  [wdqs] Depooled `wdqs2023` 
in preparation to switch it to a graph split host

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-03 Thread gerritbot 
gerritbot added a comment.


  Change #1027001 had a related patch set uploaded (by Ryan Kemper; author: 
Ryan Kemper):
  
  [operations/puppet@production] wdqs: switch wdqs2023 to graph split host
  
  https://gerrit.wikimedia.org/r/1027001

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread dr0ptp4kt 
dr0ptp4kt added a comment.


  Another thing that can be nice for figuring out stuff later is to add some 
timing and a simple log file. A command like the following was helpful when I 
was trying this out on the gaming-class desktop (you may not need this if your 
tmux session lets you scroll back really far, but it's kind of nice for tailing 
even without tmux).
  
date | tee loadData.log; time ./loadData.sh -n wdq -d 
/mnt/firehose/split_0/nt_wd_schol -s 0 -e 0 2>&1 | tee -a loadData.log; time 
./loadData.sh -n wdq -d /mnt/firehose/split_0/nt_wd_schol 2>&1 | tee -a 
loadData.log

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread dr0ptp4kt 
dr0ptp4kt added a comment.


  @RKemper I think that's captured in P54284 
 . If you need to get a copy of the 
files, there's a pointer in T350106#9381611 
 for how one might go about 
copying from HDFS to the local filesystem and then there's other stuff in the 
rest of the ticket about the data transfer. I kept a copy of the files at 
`stat1006:/home/dr0ptp4kt/gzips/nt_wd_schol` so those should be ready to be 
copied over if that helps at all.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread RKemper 
RKemper added a subscriber: dcausse.
RKemper added a comment.


  I'm realizing I don't remember enough about how we load specific graph splits 
(scholarly vs main). But it's possible we won't need the above nfs patch if our 
previous process was to manually download the relevant dump file.
  
  @dcausse We want to do a data reload of the scholarly graph for a WDQS host 
like `wdqs2023`. What's the process for that again?

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: dcausse, RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, 
Isabelladantes1983, Themindcoder, Adamm71, S8321414, Jersione, Hellket777, 
LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, 
Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, 
Akuckartz, Dringsim, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread gerritbot 
gerritbot added a comment.


  Change #1026668 had a related patch set uploaded (by Ryan Kemper; author: 
Ryan Kemper):
  
  [operations/puppet@production] wdqs: enable nfs data reloads on wdqs1021
  
  https://gerrit.wikimedia.org/r/1026668

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-02 Thread RKemper 
RKemper added a comment.


  Actually we'll use `wdqs1021` since we're not confident NFS will work 
seamlessly between eqiad and codfw (we use nsf to procure the dumps that the 
data-reload is run from).
  
  Gearing up to kick off a data reload on `wdqs1021` after a quick patch to 
enable NFS on it

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-05-01 Thread RKemper 
RKemper added a comment.


  We'll use wdqs2023 to compare against wdqs1023 (same hardware). 1023 is 
scholarly-articles so that's what we'll want to load 2023 with.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Dringsim, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-04-23 Thread Gehel 
Gehel triaged this task as "Low" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: bking, Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Dringsim, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-04-18 Thread dr0ptp4kt 
dr0ptp4kt added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: Aklapper, dr0ptp4kt, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T362920: Benchmark Blazegraph import with increased buffer capacity (and other factors)

2024-04-18 Thread dr0ptp4kt 
dr0ptp4kt renamed this task from "Benchmark Blazegraph import with increased 
buffer capacity" to "Benchmark Blazegraph import with increased buffer capacity 
(and other factors)".

TASK DETAIL
  https://phabricator.wikimedia.org/T362920

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dr0ptp4kt
Cc: Aklapper, dr0ptp4kt, AWesterinen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, KimKelting, merbst, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org