[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-28 Thread Gehel 
Gehel closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, Gehel
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-10 Thread Maintenance_bot 
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, Maintenance_bot
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Alter-paule, Beast1978, 
Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, 
Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-10 Thread Zbyszko 
Zbyszko added a comment.


  @RKemper we should be able to retry categories reload after deploying this.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-10 Thread gerritbot 
gerritbot added a comment.


  Change 625915 **merged** by jenkins-bot:
  [wikidata/query/rdf@master] Category reload shouldn't stop for single 
category fail
  
  https://gerrit.wikimedia.org/r/625915

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, gerritbot
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-10 Thread Zbyszko 
Zbyszko added a comment.


  I went with the fix for wikidata/query/rdf scripts - it made most sense to 
me, since issues with a single wiki should block updates for others. Once it's 
merged, I'll add an entry on that to runbook

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-08 Thread gerritbot 
gerritbot added a comment.


  Change 625915 had a related patch set uploaded (by ZPapierski; owner: 
ZPapierski):
  [wikidata/query/rdf@master] Category reload shouldn't stop for single 
category fail
  
  https://gerrit.wikimedia.org/r/625915

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, gerritbot
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-08 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, gerritbot
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-09-03 Thread Zbyszko 
Zbyszko claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-08-31 Thread CBogen 
CBogen set the point value for this task to "3".

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: CBogen
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-08-31 Thread Gehel 
Gehel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-08-24 Thread dcausse 
dcausse added a comment.


  There are two dump types involved here:
  
  - the full dumps happening once a week and generating a RDF turtle file 
(puppet: `modules/snapshot/files/cron/dumpcategoriesrdf.sh`)
  - the daily diffs generating a SPARQL update file (puppet: 
`modules/snapshot/files/cron/dumpcategoriesrdf-daily.sh`)
  
  to know the date boundaries to run the daily dumps the shell scripts relies 
on timestamps stored in these `last` files. I suspect some erroneous conditions 
on the timestamps that prevent the daily dump from running.
  
  The `thankyou` wiki is pretty new and is probably why it's failing just now.
  
  There are a couple of quick fixes that could be made I suppose:
  
  - fix `modules/snapshot/files/cron/dumpcategoriesrdf-daily.sh` so that it 
supports the first run and properly generate a `last` file
  - on the wdqs side (`wikidata/query/rdf`: 
`dist/src/script/loadCategoryDaily.sh`) the daily reload should not fail on a 
missing `last` file
  - the list of wikis of wikis to dump and import is seeded from the the 
categories-rdf dblist (`operations/mediawiki-config` repo), this wiki 
`thankyou` wiki is perhaps not needed here
  
  Rewriting all this machinery with a higher level language is a large 
undertaking but definitely worthwhile as it's causing a lot of pain to debug 
and is quite fragile currently.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-08-24 Thread Gehel 
Gehel triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T261097: WDQS Categories reload is failing on thankyouwiki

2020-08-24 Thread Gehel 
Gehel created this task.
Gehel added projects: Wikidata-Query-Service, Operations, Discovery-Search 
(Current work).
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  As a client of the Categories SPARQL endpoint, I want up to date data.
  
  Categories update are failing on all WDQS servers, update lag is > 8 days. A 
quick look at the logs show:
  
>Processing thankyouwiki...
Could not load timestamp from 
https://dumps.wikimedia.org/other/categoriesrdf/daily/lastdump/thankyouwiki-daily.last
2020-08-24T07:30:56+00:00 categories daily load done
  
  That dump isn't available on the latest dumps 
.
  
  While we have an entry about categories lag in the WDQS runbook 
,
 it is limited and does not seem to apply here. We need to have a more robust 
mechanism for categories update, maybe porting the current bash script to a 
higher level language.
  
  AC:
  
  [ ] categories are up to date on all WDQS servers
  [ ] categories update mechanism is robust enough
  [ ] if there is still a known case where manual intervention is required, 
this case is documented in the WDQS Runbook 


TASK DETAIL
  https://phabricator.wikimedia.org/T261097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, lmata, CBogen, Akuckartz, darthmon_wmde, 
Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs