[Wikidata-bugs] [Maniphest] T301461: Investigate cache issues after WDQS UI deployments

2022-06-21 Thread RKemper
RKemper added a comment.


  In T301461#8014754 <https://phabricator.wikimedia.org/T301461#8014754>, 
@Lucas_Werkmeister_WMDE wrote:
  
  > @RKemper thanks, that would be great!
  
  Sorry for the delay here, we had some confusion as to whether the correct 
file was modified in the previous patch (spoiler alert: it was!). Just reloaded 
the config now and tests are now passing after a quick patch 
<https://gerrit.wikimedia.org/r/c/operations/puppet/+/807200>:
  
ryankemper@cumin1001:~$ httpbb 
/srv/deployment/httpbb-tests/query_service/test_wdqs.yaml --hosts 
wdqs1012.eqiad.wmnet
Sending to wdqs1012.eqiad.wmnet...
PASS: 4 requests sent to wdqs1012.eqiad.wmnet. All assertions passed.
  
  We should be good here, although I'll defer to @Lucas_Werkmeister_WMDE for 
any further testing

TASK DETAIL
  https://phabricator.wikimedia.org/T301461

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, EBernhardson, RLazarus, ItamarWMDE, toan, Lucas_Werkmeister_WMDE, 
Aklapper, Fernandobacasegua34, Astuthiodit_1, AWesterinen, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, EBjune, 
merbst, LawExplorer, Salgo60, Lewizho99, Maathavan, _jensen, rosalieper, 
Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301461: Investigate cache issues after WDQS UI deployments

2022-06-10 Thread RKemper
RKemper added a comment.


  In T301461#7994256 <https://phabricator.wikimedia.org/T301461#7994256>, 
@Lucas_Werkmeister_WMDE wrote:
  
  > Status update: the Puppet change has been merged, but apparently the Apache 
config isn’t reloaded automatically. @RLazarus indicated on Gerrit that 
@EBernhardson would be the right person for that.
  
  @Lucas_Werkmeister_WMDE thanks for the heads up, I'll get the configs 
reloaded on Monday.

TASK DETAIL
  https://phabricator.wikimedia.org/T301461

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, EBernhardson, RLazarus, ItamarWMDE, toan, Lucas_Werkmeister_WMDE, 
Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
GoranSMilovanovic, Mahir256, QZanden, EBjune, merbst, LawExplorer, Salgo60, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

2022-05-20 Thread RKemper
RKemper added a comment.


  @Dominicbm We made `the error message is understandable by users` part of the 
acceptance criteria of this ticket, so I think that's why T306919 
<https://phabricator.wikimedia.org/T306919> was closed as a duplicate. 
Basically this ticket covers both the fixing of the actual problem as well as 
seeing what can be done to make the error message itself more comprehensible.

TASK DETAIL
  https://phabricator.wikimedia.org/T306899

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, RKemper
Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, 
Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

2022-05-19 Thread RKemper
RKemper added a comment.


  @Dominicbm @Fuzheado Understood that you all are still seeing these failures. 
We're still having trouble reproducing it on our end (we don't have great 
intuition as why but it's possible that the issue might only surface based on 
the geoDNS routing). We've added some further logging; next time any of you run 
into this issue if you could drop by the wikimedia-search and let us know, we 
can take a look at the logs and see if we can glean anything. Meanwhile on our 
end we'll keep trying to hone in on the source of the failure.

TASK DETAIL
  https://phabricator.wikimedia.org/T306899

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, RKemper
Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, 
Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-07 Thread RKemper
RKemper removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, RKemper
Cc: RKemper, bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 
Fernandobacasegua34, 786, Suran38, Biggs657, Lalamarie69, Juan90264, 
Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, 
Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan, 
Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-07 Thread RKemper
RKemper reopened this task as "In Progress".
RKemper added a comment.


  Oops, I just meant to move on workboard, not sure how I closed it as well

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, RKemper
Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-07 Thread RKemper
RKemper closed this task as "Resolved".
RKemper moved this task from Needs review to Waiting on the Discovery-Search 
(Current work) board.
RKemper added a comment.


  Moving to `Waiting` while we see how the newest settings do

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, RKemper
Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-07 Thread RKemper
RKemper added a comment.


  To check for presence of touched file:
  
ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-public' '[ -f 
"/tmp/wdqs_blazegraph_jvmquake_warn_gc" ] && echo yes || echo no'
11 hosts will be targeted:
wdqs[2001-2004,2007].codfw.wmnet,wdqs[1004-1007,1012-1013].eqiad.wmnet
Ok to proceed on 11 hosts? Enter the number of affected hosts to confirm or 
"q" to quit 11
= NODE GROUP =
(5) wdqs[2001-2002,2007].codfw.wmnet,wdqs[1004-1005].eqiad.wmnet
- OUTPUT of '[ -f "/tmp/wdqs_...o yes || echo no' -
no
= NODE GROUP =
(6) wdqs[2003-2004].codfw.wmnet,wdqs[1006-1007,1012-1013].eqiad.wmnet
- OUTPUT of '[ -f "/tmp/wdqs_...o yes || echo no' -
yes

PASS |███| 100% (11/11) 
[00:00<00:00, 12.38hosts/s]
FAIL ||   
0% (0/11) [00:00= 100.0% threshold) for command: '[ -f 
"/tmp/wdqs_...o yes || echo no'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
  
  To clear away current touched files:
  
ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-public' "rm -fv 
'/tmp/wdqs_blazegraph_jvmquake_warn_gc'"
11 hosts will be targeted:
wdqs[2001-2004,2007].codfw.wmnet,wdqs[1004-1007,1012-1013].eqiad.wmnet
Ok to proceed on 11 hosts? Enter the number of affected hosts to confirm or 
"q" to quit 11
= NODE GROUP =
(6) wdqs[2003-2004].codfw.wmnet,wdqs[1006-1007,1012-1013].eqiad.wmnet
- OUTPUT of 'rm -fv '/tmp/wdq...vmquake_warn_gc'' -
removed '/tmp/wdqs_blazegraph_jvmquake_warn_gc'

PASS |███| 100% (11/11) 
[00:00<00:00, 12.55hosts/s]
FAIL ||   
0% (0/11) [00:00= 100.0% threshold) for command: 'rm -fv 
'/tmp/wdq...vmquake_warn_gc''.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, RKemper
Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, 
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad

2022-03-15 Thread RKemper
RKemper moved this task from Waiting to Needs Reporting on the Discovery-Search 
(Current work) board.
RKemper added a comment.


  Alert appears to have resolved after T301108 
<https://phabricator.wikimedia.org/T301108>
  
  F35007412: image.png <https://phabricator.wikimedia.org/F35007412>

TASK DETAIL
  https://phabricator.wikimedia.org/T302830

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Aklapper, RKemper, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301650: WCQS "Application Connection Error" E009

2022-03-10 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  F34997527: image.png <https://phabricator.wikimedia.org/F34997527>
  
  New `cache-control` header is visible in Chrome and Firefox now. We're 
hopeful that this should fix the problem, but if not we won't spend too much 
more time digging in.
  
  @Dominicbm Let us know if you still experience this behavior in the future. 
Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T301650

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, MPhamWMF, EBernhardson, Zbyszko, Aklapper, Dominicbm, 
Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, 
Invadibot, Lalamarie69, maantietaja, FRomeo_WMF, Juan90264, Alter-paule, 
Beast1978, CBogen, Un1tY, Nintendofan885, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, JKSTNK, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad

2022-03-01 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302830

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad

2022-03-01 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302830

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad

2022-03-01 Thread RKemper
RKemper triaged this task as "High" priority.
RKemper added a project: Wikidata-Query-Service.

TASK DETAIL
  https://phabricator.wikimedia.org/T302830

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

2022-02-24 Thread RKemper
RKemper added a project: SRE-OnFire.

TASK DETAIL
  https://phabricator.wikimedia.org/T238751

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe, RKemper
Cc: Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, Addshore, Aklapper, 
Fernandobacasegua34, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, 
Lalamarie69, LSobanski, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, jijiki, 
Klaas_Z4us_V, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, elukey, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Jay8g
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301953: Investigate wdqs1013 stability issues

2022-02-23 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301953

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Zbyszko, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301167: Service implementation for wdqs20[09,10,11,12]

2022-02-07 Thread RKemper
RKemper created this task.
RKemper added projects: Discovery-Search, Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  Procurement by dc-ops done in T291982; racking by dc-ops done in T294297 
<https://phabricator.wikimedia.org/T294297>
  
  AC
  
  [ ] 4 new hosts (3 refresh, 1 expansion), `wdqs20[09,10,11,12]` brought into 
service
  [ ] wdqs200[1-3] decom'd after new hosts brought into service
  
  Note
  
  - 
https://docs.google.com/spreadsheets/d/1EYK0x4GCxv1fG77Co-S-uBiPqrvXRiLzIW34kOAJWws/edit#gid=342761434
 shows wdqs2001 and wdqs2002 should be decom'd ASAP to reduce power draw in the 
DCs. So this ticket will take care of that (given that decom'ing the old hosts 
is part of the AC)

TASK DETAIL
  https://phabricator.wikimedia.org/T301167

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-07 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Michael, Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T258754: Define SLOs and error budget for WDQS

2022-01-31 Thread RKemper
RKemper added a comment.


  Ryan will add a quick blurb in 
https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service about the SLO with a 
link to the dashboard, at which point we can move this to `Needs Reporting`.
  
  Also we (Ryan/Brian) should mention the SLO in our next SRE meeting.

TASK DETAIL
  https://phabricator.wikimedia.org/T258754

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, Addshore, Gehel, RKemper, Aklapper, bking, Invadibot, 
maantietaja, CBogen, Akuckartz, ET4Eva, Nandana, Namenlos314, jijiki, 
Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, Avner, elukey, _jensen, rosalieper, Scott_WUaS, 
Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Jay8g
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T258754: Define SLOs and error budget for WDQS

2022-01-31 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T258754

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, Addshore, Gehel, RKemper, Aklapper, bking, Invadibot, 
maantietaja, CBogen, Akuckartz, ET4Eva, Nandana, Namenlos314, jijiki, 
Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, Avner, elukey, _jensen, rosalieper, Scott_WUaS, 
Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Jay8g
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T264181: Migrate WDQS to profile::java

2021-12-11 Thread RKemper
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search 
Platform team.
Restricted Application added a project: wdwb-tech.

TASK DETAIL
  https://phabricator.wikimedia.org/T264181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, MoritzMuehlenhoff, joanna_borun, Invadibot, MPhamWMF, Devnull, 
maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, Legado_Shulgin, 
ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, 
LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, 
Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T193473: Add HTTPS support to wdqs-internal service

2021-12-11 Thread RKemper
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search 
Platform team.

TASK DETAIL
  https://phabricator.wikimedia.org/T193473

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Smalyshev, Gehel, joanna_borun, Invadibot, MPhamWMF, Devnull, 
maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, ET4Eva, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, 
FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T270614: Automatically depool wdqs servers that are "lagged"

2021-12-11 Thread RKemper
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search 
Platform team.

TASK DETAIL
  https://phabricator.wikimedia.org/T270614

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: So9q, Addshore, Moebeus, Nikki, Mahir256, Lydia_Pintscher, Aklapper, 
dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T193473: Add HTTPS support to wdqs-internal service

2021-12-07 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T193473

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Smalyshev, Gehel, joanna_borun, Invadibot, MPhamWMF, Devnull, 
maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, ET4Eva, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, 
FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

2021-11-08 Thread RKemper
RKemper added a comment.


  In T294961#7489817 <https://phabricator.wikimedia.org/T294961#7489817>, 
@Gehel wrote:
  
  > Let's make sure the kernel version is pinned somewhere in our puppet code! 
Then we can wait to see if the problem is reproduced or not.
  
  
  
  >  ryankemper, ebernhardson: yeah, let's install the kernel package 
manually for testing and if it fixes the issue, then we can apply the 
profile::base::linux510
  
  So specifically we'll apply `profile::base::linux510`

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

2021-11-08 Thread RKemper
RKemper merged a task: T294865: wcqs1002 and wcqs2001 unresponsive.
RKemper added a subscriber: Dzahn.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive

2021-11-08 Thread RKemper
RKemper closed this task as a duplicate of T294961: Resolve kernel hang on 
wcqs* instances.

TASK DETAIL
  https://phabricator.wikimedia.org/T294865

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, RKemper, Gehel, Aklapper, 786, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, 
Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

2021-11-03 Thread RKemper
RKemper added a comment.


  DNS changes from https://gerrit.wikimedia.org/r/c/operations/dns/+/736585
  -
  
ryankemper@authdns1001:~$ sudo -i authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 2c9042d86630ef640aaf1ab101c9f98bda1824e0...

 templates/wmnet| 1 -
 utils/mock_etc/discovery-geo-resources | 1 -
 2 files changed, 2 deletions(-)

diff --git templates/wmnet templates/wmnet
index c89c1610..041fc7b9 100644
--- templates/wmnet
+++ templates/wmnet
@@ -654,7 +654,6 @@ push-notifications 300/10 IN DYNA 
geoip!disc-push-notifications
 recommendation-api 300/10 IN DYNA geoip!disc-recommendation-api
 wdqs   300/10 IN DYNA geoip!disc-wdqs
 wdqs-internal  300/10 IN DYNA geoip!disc-wdqs-internal
-wcqs   300/10 IN DYNA geoip!disc-wcqs
 zotero 300/10 IN DYNA geoip!disc-zotero
 blubberoid 300/10 IN DYNA geoip!disc-blubberoid
 eventgate-analytics 300/10 IN DYNA geoip!disc-eventgate-analytics
diff --git utils/mock_etc/discovery-geo-resources 
utils/mock_etc/discovery-geo-resources
index 64e612b3..41a50fc4 100644
--- utils/mock_etc/discovery-geo-resources
+++ utils/mock_etc/discovery-geo-resources
@@ -66,4 +66,3 @@ disc-shellbox-media  => { map => mock, dcmap => { 
mock => 192.0.2.1 } }
 disc-shellbox-syntaxhighlight => { map => mock, dcmap => { mock => 
192.0.2.1 } }
 disc-shellbox-timeline   => { map => mock, dcmap => { mock => 192.0.2.1 } }
 disc-tegola-vector-tiles => { map => mock, dcmap => { mock => 192.0.2.1 } }
-disc-wcqs=> { map => mock, dcmap => { mock => 192.0.2.1 } }

Merge these changes? (yes/no)? yes
Updating 7a86417b..2c9042d8
Fast-forward
 templates/wmnet| 1 -
 utils/mock_etc/discovery-geo-resources | 1 -
 2 files changed, 2 deletions(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.oenx7otf
 -- Generating zonefiles from zone templates
 -- Processed 211 zones into directory /tmp/dns-check.oenx7otf/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 37
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 65
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 132 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.oenx7otf
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.oenx7otf to system dirs
 -- Zone changed: wmnet
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive

2021-11-03 Thread RKemper
RKemper added a subtask: T294961: Resolve kernel hang on wcqs* instances.

TASK DETAIL
  https://phabricator.wikimedia.org/T294865

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Suran38, Biggs657, joanna_borun, Invadibot, 
Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, Muchiri124, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, RhinosF1, 
joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

2021-11-03 Thread RKemper
RKemper added a parent task: T294865: wcqs1002 and wcqs2001 unresponsive.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T260568: [EPIC] Productionize WCQS

2021-11-03 Thread RKemper
RKemper added a subtask: T294865: wcqs1002 and wcqs2001 unresponsive.

TASK DETAIL
  https://phabricator.wikimedia.org/T260568

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: GFontenelle_WMF, So9q, Fuzheado, Zache, ttaylor, CBogen, Gehel, Aklapper, 
Suran38, Invadibot, MPhamWMF, maantietaja, Peteosx1x, NavinRizwi, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive

2021-11-03 Thread RKemper
RKemper added a parent task: T260568: [EPIC] Productionize WCQS.

TASK DETAIL
  https://phabricator.wikimedia.org/T294865

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Suran38, Biggs657, joanna_borun, Invadibot, 
Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, Muchiri124, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, RhinosF1, 
joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production

2021-10-18 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T288231

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: elukey, Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, 
Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293638: Create and update the process for separate WCQS mutation topic

2021-10-18 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T293638

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Ottomata, Aklapper, Zbyszko, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production

2021-10-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T288231

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, 
Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production

2021-10-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T288231

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, 
Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production

2021-10-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T288231

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, 
Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production

2021-10-14 Thread RKemper
RKemper added a comment.


  [data-transfer cookbook -> kafka offset output]
  ---
  
  (including this output since it's the first time we've relied on the new 
kafka offset part of the data-transfer process; won't be posting the output for 
any subsequent cookbooks runs)
  
  Also note that the output is a little tricky to parse. I can't tell if it did 
succeed nad the `Error sending OffsetCommitRequest_v2...` log lines were errors 
that were resolved upon retry, or if the cookbook falsely reported the offset. 
But judging by the fact that `wdqs2005` is reporting the correct number of 
triples, I have to assume that it worked properly. (This also lines up with the 
fact that we do have log lines that seem to report success: `Offsets set for 
target cluster "main", site "codfw" and consumer group "wdqs2005"`)
  
Transferring Kafka offsets
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to 
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0) 
IPv6]
Probing node bootstrap-2 broker version
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection 
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests 
on startup
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to 
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0) 
IPv6]
Probing node bootstrap-1 broker version
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection 
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests 
on startup
Same cluster, setting offsets...
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to 
kafka-main2004.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
 [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: connecting to 
kafka-main2001.codfw.wmnet:9093 [('2620:0:860:101:10:192:0:17', 9093, 0, 0) 
IPv6]
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection 
complete.
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing 
connection.
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to 
kafka-main2005.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:46', 9093, 0, 0) IPv6]
Group coordinator for wdqs2008 is BrokerMetadata(nodeId='coordinator-2003', 
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2008
 [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Connection 
complete.
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to 
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0) 
IPv6]
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection 
complete.
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
Extracted offsets from source cluster "main", site "codfw" and consumer 
group "wdqs2008".
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to 
kafka-main2005.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:46', 9093, 0, 0) 
IPv6]
Group coordinator for wdqs2005 is BrokerMetadata(nodeId='coordinator-2003', 
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2005
Error sending OffsetCommitRequest_v2 to node coordinator-2003 
[NodeNotReadyError: coordinator-2003]
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to 
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0) 
IPv6]
Error sending OffsetCommitRequest_v2 to node coordinator-2003 
[NodeNotReadyError: coordinator-2003]
Error sending OffsetCommitRequest_v2 to node coordinator-2003 
[NodeNotReadyError: coordinator-2003]
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection 
complete.
Error sending OffsetCommitRequest_v2 to node coordinator-2003 
[NodeNotReadyError: coordinator-2003]
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing 
connection.
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing 
connection.
Offsets set for target cluster "main", site "codfw" and consumer group 
"wdqs2005".
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
 [IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing 
connection.
 [IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Closing 
connection.
 [IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing 
connection.
 [IPv6 
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
  

[Wikidata-bugs] [Maniphest] T282117: WCQS needs to be exposed through a wikimedia.org domain

2021-09-28 Thread RKemper
RKemper added a comment.


  [Log of failed authdns-update following https://gerrit.wikimedia.org/r/724520]
  --
  
ryankemper@authdns1001:~$ sudo -i authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing c81f47118e6e6fea77cc8970885dd61b61c92bd0...

 templates/wmnet| 1 +
 utils/mock_etc/discovery-geo-resources | 1 +
 2 files changed, 2 insertions(+)

diff --git templates/wmnet templates/wmnet
index ea3d7279..9d554061 100644
--- templates/wmnet
+++ templates/wmnet
@@ -601,6 +601,7 @@ push-notifications 300/10 IN DYNA 
geoip!disc-push-notifications
 recommendation-api 300/10 IN DYNA geoip!disc-recommendation-api
 wdqs   300/10 IN DYNA geoip!disc-wdqs
 wdqs-internal  300/10 IN DYNA geoip!disc-wdqs-internal
+wcqs   300/10 IN DYNA geoip!disc-wcqs
 zotero 300/10 IN DYNA geoip!disc-zotero
 blubberoid 300/10 IN DYNA geoip!disc-blubberoid
 eventgate-analytics 300/10 IN DYNA geoip!disc-eventgate-analytics
diff --git utils/mock_etc/discovery-geo-resources 
utils/mock_etc/discovery-geo-resources
index d5f686e1..0d51b0bc 100644
--- utils/mock_etc/discovery-geo-resources
+++ utils/mock_etc/discovery-geo-resources
@@ -65,3 +65,4 @@ disc-shellbox-media  => { map => mock, dcmap => { 
mock => 192.0.2.1 } }
 disc-shellbox-syntaxhighlight => { map => mock, dcmap => { mock => 
192.0.2.1 } }
 disc-shellbox-timeline   => { map => mock, dcmap => { mock => 192.0.2.1 } }
 disc-tegola-vector-tiles => { map => mock, dcmap => { mock => 192.0.2.1 } }
+disc-wcqs=> { map => mock, dcmap => { mock => 192.0.2.1 } }

Merge these changes? (yes/no)? yes
Updating e1e708d9..c81f4711
Fast-forward
 templates/wmnet| 1 +
 utils/mock_etc/discovery-geo-resources | 1 +
 2 files changed, 2 insertions(+)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.uhizg7cf
 -- Generating zonefiles from zone templates
 -- Processed 211 zones into directory /tmp/dns-check.uhizg7cf/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 37
W002|MISSING_PTR_FOR_NAME_AND_IP: 26
W103|MISSING_MGMT_FOR_NAME: 64
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.uhizg7cf
Traceback (most recent call last):
  File "utils/deploy-check.py", line 276, in 
main()
  File "utils/deploy-check.py", line 268, in main
deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir), 
gdir)
  File "utils/deploy-check.py", line 214, in deploy_check
safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
  File "utils/deploy-check.py", line 88, in safe_cmd
p_err.decode('utf-8')))
Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.uhizg7cf checkconf 
failed with exit code 42, stderr:
info: gdnsd version 3.8.0 @ pid 22926
info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
info: plugin_geoip: map 'generic-map': Loading GeoIP2 database 
'/tmp/dns-check.uhizg7cf/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: 
GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-27 17:10:01 UTC
info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1122136 
dclists: 8
info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410 
dclists: 2
info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database 
'/tmp/dns-check.uhizg7cf/geoip/GeoIP2-City.mmdb': Version: 2.0, Type: 
GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-27 17:10:01 UTC
info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464 
dclists: 2
info: admin_state: checking state file 
'/tmp/dns-check.uhizg7cf/state/admin_state'...
error: plugin_geoip: In

[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment

2021-09-28 Thread RKemper
RKemper added a comment.


~/wmf/rdf/queries [master!]% ./test.sh -s http://localhost:/
Running 1592.sparqlcurl -G --data-urlencode query@1592.sparql --data 
format=json --data random=62170 http://localhost://sparql -s

 PASSED
$KEEP is no, nuked /tmp/querytest.1592.62170
Running 1785.sparqlcurl -G --data-urlencode query@1785.sparql --data 
format=json --data random=62170 http://localhost://sparql -s

 PASSED
$KEEP is no, nuked /tmp/querytest.1785.62170
Running badservice.sparqlcurl -G --data-urlencode query@badservice.sparql 
--data format=json --data random=62170 http://localhost://sparql -s

 FAILED
$KEEP is no, nuked /tmp/querytest.badservice.62170
Running cats.sparqlcurl -G --data-urlencode query@cats.sparql --data 
format=json --data random=62170 http://localhost://sparql -s

 PASSED
$KEEP is no, nuked /tmp/querytest.cats.62170
Running dates.sparqlcurl -G --data-urlencode query@dates.sparql --data 
format=json --data random=62170 http://localhost://sparql -s

 PASSED
$KEEP is no, nuked /tmp/querytest.dates.62170
Running minus.sparqlcurl -G --data-urlencode query@minus.sparql --data 
format=json --data random=62170 http://localhost://sparql -s
  
  Test queries passed during deploy of 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/720801 to canary `wdqs1003`

TASK DETAIL
  https://phabricator.wikimedia.org/T280247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, RKemper
Cc: WMDE-leszek, RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, 
ttaylor, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, 
Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-09-23 Thread RKemper
RKemper added a comment.


  [Steps for deploy of https://gerrit.wikimedia.org/r/723254]
  ---
  
merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/723254
(on cumin) => sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'
ack alerts

!log T280001 Restarting pybal on backup low-traffic hosts `lvs2010` and 
`lvs1016`
sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'


Sanity check of `sudo ipvsadm -L -n` on low-traffic backups `lvs2010` and 
`lvs1016` => contains wcqs and has sane list of backends (`wcqs*`)

!log T280001 Sanity check of `sudo ipvsadm -L -n` on backup  `lvs2010` and 
`lvs1016` looks good, proceeding

Wait 120s while checking https://icinga.wikimedia.org/alerts
!log Waited 120s and checked https://icinga.wikimedia.org/alerts, 
proceeding to primary low-traffic hosts `lvs2009` and `lvs1015`

!log T280001 Restarting pybal on low-traffic primaries `lvs2009` and 
`lvs1015`
sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'

Run a test like `curl -v -k http://wcqs.svc.eqiad.wmnet`
!log T280001 Sanity check of `wcqs.svc.{eqiad,codfw}.wmnet` looks good; 
declaring this a success

TASK DETAIL
  https://phabricator.wikimedia.org/T280001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-09-23 Thread RKemper
RKemper added a comment.


  [conftool changes following merge of 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959]
  
-
  
Now running conftool-merge to sync any changes to conftool data
Running conftool-sync on /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity 
node from /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/node/codfw.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/node/eqsin.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/node/eqiad.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/node/esams.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/node/ulsfo.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity 
discovery from /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/discovery/services.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/discovery/mediawiki.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity 
mwconfig from /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/mwconfig/data.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity 
dbconfig-instance from /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/dbconfig-instance/instances.yaml
2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity 
dbconfig-section from /etc/conftool/data
2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file 
/etc/conftool/data/dbconfig-section/sections.yaml
2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for node
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
codfw/wcqs/wcqs/wcqs2003.codfw.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
codfw/wcqs/wcqs/wcqs2002.codfw.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
eqiad/wcqs/wcqs/wcqs1001.eqiad.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
eqiad/wcqs/wcqs/wcqs1003.eqiad.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
eqiad/wcqs/wcqs/wcqs1002.eqiad.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags 
codfw/wcqs/wcqs/wcqs2001.codfw.wmnet
2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for discovery
2021-09-23 16:38:10 [INFO] conftool::load: Creating discovery with tags 
wcqs/eqiad
2021-09-23 16:38:10 [INFO] conftool::load: Creating discovery with tags 
wcqs/codfw
2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for mwconfig
2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for 
dbconfig-instance
2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for 
dbconfig-section
2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for 
dbconfig-section
2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for 
dbconfig-instance
2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for 
mwconfig
2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for 
discovery
2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for node

TASK DETAIL
  https://phabricator.wikimedia.org/T280001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-09-21 Thread RKemper
RKemper added a comment.


  [DNS change -> sre.dns.netbox cookbook]
  ---
  
ryankemper@cumin1001:~$ sudo -i cookbook sre.dns.netbox -t T280001 'Added 
wcqs.svc.[eqiad,codfw].wmnet'
START - Cookbook sre.dns.netbox
Generating the DNS records from Netbox data. It will take a couple of 
minutes.
- OUTPUT of 'cd /tmp && runus...ad,codfw].wmnet"' -
2021-09-21 18:57:04,823 [INFO] Gathering devices, interfaces, addresses and 
prefixes from Netbox
2021-09-21 19:00:22,214 [WARNING] Device frqueue1002 of IP 10.64.40.204/26 
not in devices, skipping.
2021-09-21 19:00:22,219 [WARNING] Device phab1003 of IP 10.65.1.16/16 not 
in devices, skipping.
2021-09-21 19:00:22,262 [INFO] Gathered 2307 devices from Netbox
2021-09-21 19:00:22,262 [INFO] Generating DNS records
2021-09-21 19:00:32,939 [INFO] Generated 13348 direct and reverse records 
(6674 each) in 28 direct zones and 171 reverse zones
2021-09-21 19:00:32,940 [INFO] Cloning /srv/netbox-exports/dns.git/ to 
/tmp/dns-c25pcHBldHM-5qrdksyv ...
2021-09-21 19:00:33,215 [INFO] Generating zonefile snippets to directory 
/tmp/dns-c25pcHBldHM-5qrdksyv
2021-09-21 19:00:34,249 [INFO] Committed changes: 
4a37a231dbab6bfb2f424feffef4d4f411b8dd49
2021-09-21 19:00:34,285 [INFO] Validating generated data
2021-09-21 19:00:34,286 [INFO] Commit details: {'insertions': 4, 
'deletions': 0, 'lines': 4, 'files': 4}
commit 4a37a231dbab6bfb2f424feffef4d4f411b8dd49
Author: generate-dns-snippets 
Date:   Tue Sep 21 19:00:34 2021 +

ryankemper@cumin1001: Added wcqs.svc.[eqiad,codfw].wmnet

diff --git a/1.2.10.in-addr.arpa b/1.2.10.in-addr.arpa
index 16f68f5..0d9297a 100644
--- a/1.2.10.in-addr.arpa
+++ b/1.2.10.in-addr.arpa
@@ -53,3 +53,4 @@
 64  1H IN PTR shellbox-media.svc.codfw.wmnet.
 65  1H IN PTR shellbox-syntaxhighlight.svc.codfw.wmnet.
 66  1H IN PTR shellbox-timeline.svc.codfw.wmnet.
+67  1H IN PTR wcqs.svc.codfw.wmnet.
diff --git a/2.2.10.in-addr.arpa b/2.2.10.in-addr.arpa
index adeb0c3..6904f40 100644
--- a/2.2.10.in-addr.arpa
+++ b/2.2.10.in-addr.arpa
@@ -58,3 +58,4 @@
 64  1H IN PTR shellbox-media.svc.eqiad.wmnet.
 65  1H IN PTR shellbox-syntaxhighlight.svc.eqiad.wmnet.
 66  1H IN PTR shellbox-timeline.svc.eqiad.wmnet.
+67  1H IN PTR wcqs.svc.eqiad.wmnet.
diff --git a/svc.codfw.wmnet b/svc.codfw.wmnet
index 23001c2..965466b 100644
--- a/svc.codfw.wmnet
+++ b/svc.codfw.wmnet
@@ -50,6 +50,7 @@ thanos-swift 1H IN A 10.2.1.54
 thumbor  1H IN A 10.2.1.24
 toolhub  1H IN A 10.2.1.62
 videoscaler  1H IN A 10.2.1.5
+wcqs 1H IN A 10.2.1.67
 wdqs 1H IN A 10.2.1.32
 wdqs-internal1H IN A 10.2.1.41
 wikifeeds1H IN A 10.2.1.47
diff --git a/svc.eqiad.wmnet b/svc.eqiad.wmnet
index 3dd0aa1..185c684 100644
--- a/svc.eqiad.wmnet
+++ b/svc.eqiad.wmnet
@@ -57,6 +57,7 @@ thanos-swift 1H IN A 10.2.2.54
 thumbor  1H IN A 10.2.2.24
 toolhub  1H IN A 10.2.2.62
 videoscaler  1H IN A 10.2.2.5
+wcqs 1H IN A 10.2.2.67
 wdqs 1H IN A 10.2.2.32
 wdqs-internal1H IN A 10.2.2.41
 wikifeeds1H IN A 10.2.2.47
METADATA: {"path": "/tmp/dns-c25pcHBldHM-5qrdksyv", "sha1": 
"4a37a231dbab6bfb2f424feffef4d4f411b8dd49", "insertions": 4, "deletions": 0, 
"lines": 4, "files": 4}

PASS 
|█|
 100% (1/1) [03:30<00:00, 210.69s/hosts]
FAIL |  



|   0% (0/1) [03:30= 100.0% threshold) for command: 'cd /tmp && 
runus...ad,codfw].wmnet"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
>>> Have you checked that the diff is OK?
   

[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-09-21 Thread RKemper
RKemper added a comment.


  [DNS change -> eqiad validation]
  
  
  ryankemper@authdns1001:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t 
any wcqs.svc.eqiad.wmnet ; done
  
  ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns0.wikimedia.org -t any 
wcqs.svc.eqiad.wmnet
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48639
  ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
  ;; WARNING: recursion requested but not available
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 1024
  ; COOKIE: 2488f3a7fd0c5bbe01d8534adfbac196 (good)
  ; OPT=11: 01 72 (".r")
  ;; QUESTION SECTION:
  ;wcqs.svc.eqiad.wmnet.  IN  ANY
  
  ;; ANSWER SECTION:
  wcqs.svc.eqiad.wmnet.   3600IN  HINFO   "RFC8482" ""
  
  ;; Query time: 0 msec
  ;; SERVER: 208.80.154.238#53(208.80.154.238)
  ;; WHEN: Tue Sep 21 18:49:13 UTC 2021
  ;; MSG SIZE  rcvd: 96
  
  ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns1.wikimedia.org -t any 
wcqs.svc.eqiad.wmnet
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43542
  ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
  ;; WARNING: recursion requested but not available
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 1024
  ; COOKIE: 4ea9c8d14a719114988252f4b7a24024 (good)
  ; OPT=11: 01 72 (".r")
  ;; QUESTION SECTION:
  ;wcqs.svc.eqiad.wmnet.  IN  ANY
  
  ;; ANSWER SECTION:
  wcqs.svc.eqiad.wmnet.   3600IN  HINFO   "RFC8482" ""
  
  ;; Query time: 0 msec
  ;; SERVER: 208.80.153.231#53(208.80.153.231)
  ;; WHEN: Tue Sep 21 18:49:13 UTC 2021
  ;; MSG SIZE  rcvd: 96
  
  ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns2.wikimedia.org -t any 
wcqs.svc.eqiad.wmnet
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1138
  ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
  ;; WARNING: recursion requested but not available
  
  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 1024
  ; COOKIE: 0e7f90b6a29651b34d8115b3a817a4cd (good)
  ; OPT=11: 01 72 (".r")
  ;; QUESTION SECTION:
  ;wcqs.svc.eqiad.wmnet.  IN  ANY
  
  ;; ANSWER SECTION:
  wcqs.svc.eqiad.wmnet.   3600IN  HINFO   "RFC8482" ""
  
  ;; Query time: 0 msec
  ;; SERVER: 91.198.174.239#53(91.198.174.239)
  ;; WHEN: Tue Sep 21 18:49:13 UTC 2021
  ;; MSG SIZE  rcvd: 96
  
  [DNS change -> codfw validation]
  
  
ryankemper@authdns1001:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t 
any wcqs.svc.codfw.wmnet ; done

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns0.wikimedia.org -t any 
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36183
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 8dad192df87e3fd3a2874c513054386d (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.codfw.wmnet.  IN  ANY

;; ANSWER SECTION:
wcqs.svc.codfw.wmnet.   3600IN  HINFO   "RFC8482" ""

;; Query time: 0 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Tue Sep 21 18:49:39 UTC 2021
;; MSG SIZE  rcvd: 96


; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns1.wikimedia.org -t any 
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10558
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 8b49e3f1d8cc32489ef6822def8f7e99 (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.codfw.wmnet.  IN  ANY

;; ANSWER SECTION:
wcqs.svc.codfw.wmnet.   3600IN  HINFO   "RFC8482" ""

;; Query time: 0 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Tue Sep 21 18:49:39 UTC 2021
;; MSG SIZE  rcvd: 96


; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns2.wikimedia.org -t any 
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- o

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread RKemper
RKemper added a comment.


  We're no longer doing hourly restarts; we should monitor availability over 
the weekend to make sure that we're capable of handling current load without 
the crutch of the hourly restarts. We still have the useragent we banned in the 
last full outage (dailymotion) banned, so I'd think we'll be okay on load, but 
I'll be checking Grafana over the next couple days.
  
  We can/should unban dailymotion early next week - perhaps tuesday - if wdqs 
has been stable over the weekend and during monday's load

TASK DETAIL
  https://phabricator.wikimedia.org/T290330

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Mholloway, 
Gehel, Ladsgroup, Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment

2021-09-08 Thread RKemper
RKemper added a comment.


  @Ladsgroup https://gerrit.wikimedia.org/r/c/operations/puppet/+/719502 Okay 
the puppet change is officially merged, so the `gui-deploy` stuff should be 
unblocked now.

TASK DETAIL
  https://phabricator.wikimedia.org/T280247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, RKemper
Cc: WMDE-leszek, RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, 
ttaylor, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, 
Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T290545: Wikidata Query Service inaccessible via GUI, only partially accessible via embedded queries

2021-09-08 Thread RKemper
RKemper added a comment.


  Config change was successfully reverted. We had to clear out the varnish 
cache with `echo 'https://query.wikidata.org/' | mwscript purgeList.php`  per 
https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging.
  
  Everything should be / is back to normal now.

TASK DETAIL
  https://phabricator.wikimedia.org/T290545

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Lydia_Pintscher, Ladsgroup, RKemper, Gnoeee, Fuzheado, Aklapper, 
Daniel_Mietchen, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T290545: Wikidata Query Service inaccessible via GUI, only partially accessible via embedded queries

2021-09-08 Thread RKemper
RKemper added a comment.


  @Lydia_Pintscher Thanks this is definitely related to that work. When I 
spot-checked query.wikidata.org after merging earlier it looks like the new 
config hadn't taken effect yet.
  
  Reverted the puppet change. Working on getting things back how they should be 
currently.
  
  (Thanks for opening this @Daniel_Mietchen)

TASK DETAIL
  https://phabricator.wikimedia.org/T290545

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Lydia_Pintscher, Ladsgroup, RKemper, Gnoeee, Fuzheado, Aklapper, 
Daniel_Mietchen, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment

2021-09-07 Thread RKemper
RKemper added a comment.


  @Ladsgroup Puppet patch is merged now. I haven't merged the gui-deploy 
patches yet so those two gui-deploy patches should be good to go whenever now 
that the puppet changes are in place

TASK DETAIL
  https://phabricator.wikimedia.org/T280247

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, RKemper
Cc: RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, ttaylor, 
Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280006: Set up the application authentication for WCQS on commons-query.wikimedia.org

2021-09-01 Thread RKemper
RKemper added a comment.


  Zbyszko, Erik, and I discussed the current status of this effort. We've 
realized that there's some open questions around how to actually implement the 
oauth, particularly with stuff like the callback URL given the way our routing 
logic works currently.
  
  Currently, for WDQS (which does not use oauth), users visit the "microsite" 
for the UI, but their requests end up hitting nginx (which forwards them to 
blazegraph).
  
  For WCQS, there won't be that pattern of a request hitting blazegraph 
directly. They'll be talking to the microsite, and the user is never going to 
do an interactive request against the backend server (e.g. nginx->blazegraph).
  
  ATS at the edge [*before* the request hits the internal loadbalancer (lvs)] 
will look at request and decide, based off the directory, whether it should go 
to the microsite or nginx. There's one special directory that will route to 
nginx, whereas everything else will go to the microsite.
  
  Where this breaks down is with oauth in the picture, this pattern won't work.
  
  - One approach would be for all traffic to route to nginx, and then have 
nginx decide what to forward to the microsite. As a general rule we were trying 
to avoid this approach though because we want this routing logic handled at the 
"infra" level, not the app layer.
  
  We'll need to figure out an approach here.

TASK DETAIL
  https://phabricator.wikimedia.org/T280006

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, hnowlan, sdkim, BPirkle, Zache, Aklapper, Gehel, CBogen, ttaylor, 
Zbyszko, Invadibot, MPhamWMF, maantietaja, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-08-31 Thread RKemper
RKemper added a comment.


  Update: `sdcquery01` has been deleted, so we can proceed with removing the 
`Optional` type

TASK DETAIL
  https://phabricator.wikimedia.org/T280001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-08-31 Thread RKemper
RKemper added a comment.


  With respect to https://gerrit.wikimedia.org/r/c/operations/puppet/+/713946, 
we'll need to know if `sdcquery01` is still in use. If it's not and we can 
"decommission" it (or at least not worry about breaking it), that will let us 
require `oauth_settings` to be defined here, instead of being an `Optional`: 
https://github.com/wikimedia/puppet/blob/ae7ad4bc10f65ce260c623a1f4c06835d26f1c52/modules/profile/manifests/query_service/wcqs.pp#L21
  
  (This is because for WCQS we always want `oauth_settings` to be set, so we 
want to disallow the use of null/undefined)

TASK DETAIL
  https://phabricator.wikimedia.org/T280001

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster

2021-08-30 Thread RKemper
RKemper added a comment.


  Log of steps to generate tls certs for wcqs with cergen for 
https://gerrit.wikimedia.org/r/713958
  
-
  
  `/srv/private` commit + 2 gerrit patches for TLS certs:
  ---
  
  (1) `[/srv/private]  Commit SHA => 
a761d97c6395aab546d0024c4fce1b7174f50aa0`
  (2) `[operations/puppet] Patch  => 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/715569`
  (3) `[labs/private]  Patch  => 
https://gerrit.wikimedia.org/r/c/labs/private/+/715570`
  
  ---
  
  Log of steps to generate tls certs for wcqs with cergen for 
https://gerrit.wikimedia.org/r/713958
  
-
  
  **(1)** Create 
`/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/wcqs.certs.yaml`
 on puppetmaster1001.eqiad.wmnet
  
  Here's what it looks like for WDQS currently:
  
ryankemper@puppetmaster1001:~$ cat 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml
wdqs.discovery.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: 
["wdqs.discovery.wmnet","wdqs.svc.eqiad.wmnet","wdqs.svc.codfw.wmnet","wdqs.wikimedia.org","wdqs1005.eqiad.wmnet","query.wikidata.org"]
  key:
password: REDACTED
algorithm: ec
  
  So let's try the following for wcqs:
  
wcqs.discovery.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: 
["wcqs.discovery.wmnet","wcqs.svc.eqiad.wmnet","wcqs.svc.codfw.wmnet","commons-query.wikimedia.org"]
  key:
password: REDACTED
algorithm: ec
  
  `sudo git add 
modules/secret/secrets/certificates/certificate.manifests.d/wcqs.certs.yaml`
  
  **(2)**
  
  `sudo cergen -c 'wcqs.*' --generate --base-path 
/srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d`
  
  `=>`
  
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wcqs.*' 
--generate --base-path /srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-08-30 17:26:10,965 INFO cergen   
Generating certificates ['wcqs.discovery.wmnet'] with force=False
2021-08-30 17:26:10,965 INFO Certificate(wcqs.discovery.wmnet)
Generating all files, force=False...
2021-08-30 17:26:10,967 INFO Certificate(wcqs.discovery.wmnet)
Generating certificate file
/usr/lib/python3/dist-packages/urllib3/connection.py:362: 
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no 
`subjectAltName`, falling back to check for a `commonName` for now. This 
feature is being removed by major browsers and deprecated by RFC 2818. (See 
https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python3/dist-packages/urllib3/connection.py:362: 
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no 
`subjectAltName`, falling back to check for a `commonName` for now. This 
feature is being removed by major browsers and deprecated by RFC 2818. (See 
https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python3/dist-packages/urllib3/connection.py:362: 
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no 
`subjectAltName`, falling back to check for a `commonName` for now. This 
feature is being removed by major browsers and deprecated by RFC 2818. (See 
https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
2021-08-30 17:26:12,621 INFO Certificate(wcqs.discovery.wmnet)
Generating CA certificate file
2021-08-30 17:26:12,622 INFO Certificate(wcqs.discovery.wmnet)
Generating PKCS12 keystore file
2021-08-30 17:26:12,951 INFO Certificate(wcqs.discovery.wmnet)
Generating Java keystore file
2021-08-30 17:26:13,931 INFO Certificate(wcqs.discovery.wmnet)
Importing PuppetCA(puppetmaster1001.eqiad.wmnet_8140) cert into Java keystore
2021-08-30 17:26:15,092 INFO Certificate(wcqs.discovery.wmnet)
Generating Java truststore file with CA certificate 
PuppetCA(puppetmaster1001.eqiad.wmnet_8140)

Status of certificates ['wcqs.discovery.wmnet']

Certificate(wcqs.discovery.wmnet, 
authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):

/srv/private/modules/secret/secrets/certificates/wcqs.discovery.wmnet/wcqs.discovery.wmnet.key.private.pem:
 PRESENT (mtime: 2021-08-30T17:26:10.966004)

/srv/private/modules/secret/secrets/certificates/wcqs.discovery.wmnet/wcqs.discovery.wmnet.key.public

[Wikidata-bugs] [Maniphest] T207676: Parallelize scap deployment of WDQS

2021-08-30 Thread RKemper
RKemper added a comment.


  I'll talk to rel-eng to see what scap changes are needed to parallelize 
between groups (wdqs eqiad public vs wdqs eqiad internal, etc)
  
  There's a chance it might be worth it to rely on a cookbook to rolling 
restart. Basically we'd use scap to get the new code in place and a cookbook to 
do the actual rolling restarts to actually uptake the changes. But for now I'd 
assume we'll just be changing it in scap-land and not introducing a cookbook

TASK DETAIL
  https://phabricator.wikimedia.org/T207676

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Smalyshev, Gehel, Mathew.onipe, Aklapper, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager

2021-08-17 Thread RKemper
RKemper added a subtask: T289077: Migrate wdqs prometheus-based alerts from 
Icinga to alert-manager.

TASK DETAIL
  https://phabricator.wikimedia.org/T281454

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Jdlrobson, fgiunchedi, Aklapper, Invadibot, MPhamWMF, maantietaja, lmata, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, herron, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288501: Blazegraph journal too large on wdqs2003

2021-08-10 Thread RKemper
RKemper claimed this task.
RKemper edited projects, added Discovery-Search (Current work); removed 
Discovery-Search.

TASK DETAIL
  https://phabricator.wikimedia.org/T288501

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, RKemper, ema, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T288501: Blazegraph journal too large on wdqs2003

2021-08-10 Thread RKemper
RKemper added a comment.


ryankemper@wdqs2003:~$ ls -lah /srv/wdqs
total 2.8T
drwxrwxr-x 3 blazegraph wikidev4.0K Aug  6 07:26 .
drwxr-xr-x 5 root   root   4.0K Jun  2 22:59 ..
-rw-rw-r-- 1 blazegraph wikidev  54 Aug  6 07:26 aliases.map
-rw-rw-r-- 1 blazegraph blazegraph  23G Aug 10 07:41 categories.jnl
-rw-r--r-- 1 root   root  0 Jun  3 01:25 data_loaded
-rw-r--r-- 1 blazegraph blazegraph  83K Aug  6 07:26 dcatap-20210806.rdf
drwxrwxr-x 2 blazegraph wikidev 68K Aug 10 07:41 dumps
-rw-rw-r-- 1 blazegraph blazegraph 2.8T Aug  9 19:25 wikidata.jnl
ryankemper@wdqs2003:~$ df -h
Filesystem  Size  Used Avail Use% Mounted on
udev 63G 0   63G   0% /dev
tmpfs13G  1.3G   12G  11% /run
/dev/md0 73G   15G   55G  22% /
tmpfs63G  4.0K   63G   1% /dev/shm
tmpfs   5.0M 0  5.0M   0% /run/lock
tmpfs63G 0   63G   0% /sys/fs/cgroup
tmpfs   1.0G   20K  1.0G   1% /var/lib/nginx
/dev/md22.9T  2.8T  120K 100% /srv
tmpfs13G 0   13G   0% /run/user/24320
  
  `wikidata.jnl` is over-inflated. here's a healthy host to compare to:
  
ryankemper@wdqs2002:~$ ls -lah /srv/wdqs
total 998G
drwxrwxr-x 3 blazegraph wikidev4.0K Aug  6 07:38 .
drwxr-xr-x 5 root   root   4.0K Jun  4 04:46 ..
-rw-rw-r-- 1 blazegraph wikidev  54 Aug  6 07:38 aliases.map
-rw-rw-r-- 1 blazegraph blazegraph  23G Aug 10 07:56 categories.jnl
-rw-r--r-- 1 root   root  0 Jun  4 06:46 data_loaded
-rw-r--r-- 1 blazegraph blazegraph  83K Aug  6 07:38 dcatap-20210806.rdf
drwxrwxr-x 2 blazegraph wikidev 64K Aug 10 07:56 dumps
-rw-rw-r-- 1 blazegraph blazegraph 975G Aug 10 17:07 wikidata.jnl
ryankemper@wdqs2002:~$ df -h
Filesystem  Size  Used Avail Use% Mounted on
udev 48G 0   48G   0% /dev
tmpfs   9.5G  952M  8.6G  10% /run
/dev/md0 73G  7.3G   62G  11% /
tmpfs48G  4.0K   48G   1% /dev/shm
tmpfs   5.0M 0  5.0M   0% /run/lock
tmpfs48G 0   48G   0% /sys/fs/cgroup
tmpfs   1.0G   20K  1.0G   1% /var/lib/nginx
/dev/md22.9T  999G  1.8T  36% /srv
tmpfs   9.5G 0  9.5G   0% /run/user/24320
  
  
  
  ---
  
  Solution is pretty simple, nuke `wikidata.jnl` (w/ the host de-pooled) and 
then kick off a `data-transfer` from `wdqs2002` (external) or `wdqs2005` 
(internal)

TASK DETAIL
  https://phabricator.wikimedia.org/T288501

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, RKemper, ema, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012

2021-06-08 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T284445

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012

2021-06-08 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T284445

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012

2021-06-08 Thread RKemper
RKemper moved this task from Ready for Development to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  This is done. New `wikidata.jnl` is 975G as expected:
  
  `-rw-rw-r-- 1 blazegraph blazegraph 975G Jun  8 19:01 wikidata.jnl`

TASK DETAIL
  https://phabricator.wikimedia.org/T284445

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper added a comment.


  Re-image of `wdqs2007` was completed successfully.

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper added a comment.


  We can't proceed any further on this ticket until we address 
https://phabricator.wikimedia.org/T280382, because the short-term fix for disk 
space will require re-imaging the affected `wdqs` hosts which will require 
starting the `data-transfer` process over (not the `data-reload`, fortunately).

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'systemctl 
start ...art wdqs-updater'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
[1/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: 'NaN'
[2/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4458.16434332'
[3/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4230.269809265'
[4/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3850.3779346'
[5/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3175.483400543'
[6/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '888.591801086'
[7/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '39.694332428'
[8/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '42.79952316284'
  
  Another small issue. Looks like we need to tell it to truncate (or just use 
doubles instead of ints)

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


  (For future reference)
  
import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
spicerack = Spicerack(verbose=True, dry_run=True)


def site(host):
"""Hacky way to get the site in which a host is located."""
if 'eqiad' in str(host):
return 'eqiad'
if 'codfw' in str(host):
return 'codfw'
raise ValueError('Site is unknown for {host}.'.format(host=host))

remote = spicerack.remote()
prometheus = spicerack.prometheus()

remote_host = remote.query('wdqs2008.codfw.wmnet')

host = remote_host.hosts[0].split(".")[0]
query = "scalar(time() - blazegraph_lastupdated{instance='%s:9193'})" % host
result = prometheus.query(query, site(remote_host))
last_updated = int(result[1])
if last_updated > 1200:
raise ValueError("Let's wait for updater to catch up, current 
last_updated of {} is too high.".format(last_updated))
  
  Snippet used to diagnose the above issue without having to actually run the 
cookbook

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper added a comment.


  Since we've seeded to `wdqs1004.eqiad.wmnet` we can start using that to 
transfer to other eqiad public nodes. Still need to seed to the other 3 (codfw 
public, codfw internal, eqiad internal) before we'll be at the peak "velocity".

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


  Tested out the new cookbook, there's at least one small issue:
  
Exception raised while executing cookbook sre.wdqs.data-transfer:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
raw_ret = runner.run()
  File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, 
in run
return self._run(self.args, self.spicerack)
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/data-transfer.py", 
line 179, in run
wait_for_updater(prometheus, site(source), source)
  File "/usr/lib/python3/dist-packages/spicerack/decorators.py", line 104, 
in wrapper
return func(*args, **kwargs)  # type: ignore
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/__init__.py", line 25, 
in wait_for_updater
last_updated = int(result['value'][1])
TypeError: list indices must be integers or slices, not str
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
  
  Error seems quite straightforward - we'll want to patch this.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper added a comment.


  I was having trouble understanding what the restart status of blazegraph has 
to do with the transfer of `/srv/wdqs/wikidata.jnl`. (As an aside, the point 
that running multiple simultaneous transfers from the same source is 
bottlenecked by outgoing bandwidth anyway makes perfect sense to me).
  
  It just clicked for me though - the cookbook transfers the file while 
blazegraph is disabled, so if another cookbook instance re-enables blazegraph 
while another cookbook instance is still receiving `/srv/wdqs/wikidata.jnl`, 
the `wikidata.jnl` will be getting mutated during the transfer.
  
  Note: I updated my comment above to indicate that `wdqs100[3,4]` are not 
properly done due to the data corruption.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper added a comment.


  [EQIAD PUBLIC]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs1004.eqiad.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1005.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1006.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1007.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1012.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1013.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [CODFW PUBLIC]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs2001.codfw.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2002.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2003.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2007.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [EQIAD INTERNAL]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs1003.eqiad.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1008.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1011.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [CODFW INTERNAL]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs2004.codfw.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2005.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2006.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nanda

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-03-24 Thread RKemper
RKemper added a comment.


  `wdqs1009`, `wdqs1010`, and `wdqs2008` are done, so we need to 
`data-transfer` to the remaining instances.
  
  It looks like `wdqs1010` is warning (not critical) for disk space on `/srv`. 
Glancing at the box it's still got 44G left but is at 96% utilization:
  
  `/dev/mapper/vg0-srv   1.1T  999G   44G  96% /srv`
  
  This is essentially entirely attributable to `wikidata.jnl` which is **1 TB** 
in size.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: ayounsi, jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, 
maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, 
Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **DNS change logs**
  
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...

 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
 ; Servers (alphabetic order)

 ; Service aliases
-query   1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.

 ; Wikis (alphabetic order)


Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
 -- Generating zonefiles from zone templates
 -- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
 -- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!
  
  Validation looks good:
  
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig 
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done

; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE  rcvd: 84

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  The issues with envoy were resolved by running `sudo 
/usr/local/sbin/build-envoy-config -c /etc/envoy` to properly build 
`/etc/envoy/envoy.yaml`. That should have been done by puppet already, 
triggered upon a `sudo systemctl restart envoyproxy.service`, but it didn't - 
perhaps a race condition. See 
https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/init.pp#81
 and 
https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/conf.pp#30
 for the puppet code that normally does it automatically.
  
  **Envoy is working properly now and wdqs1009 is now reachable from the 
caching layer**

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **DNS change logs**
  
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...

 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
 ; Servers (alphabetic order)

 ; Service aliases
-query   1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.

 ; Wikis (alphabetic order)


Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
 -- Generating zonefiles from zone templates
 -- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
 -- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!
  
  Validation looks good:
  
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig 
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done

; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE  rcvd: 84

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  I missed a step yesterday: I'd updated `/srv/private` as well as the public 
`labs/private` repo but missed the step for updating `operations/puppet` with 
the new pubkey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671267

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Current status for when I pick this back up:
  
  - `query-preview.wikidata.org` (with `wdqs1009.eqiad.wmnet` as an `alt_name`) 
is generated
  
  - reverted https://gerrit.wikimedia.org/r/c/operations/puppet/+/670339/ for 
now, envoy wasn't happy:
  
Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: Failed to call 
refresh: '/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 
instead of one of [0]
Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: 
'/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 instead of one 
of [0]

Error: 
/Stage[main]/Profile::Tlsproxy::Envoy/Sslcert::Certificate[query-preview.wikidata.org]/File[/etc/ssl/localcerts/query-preview.wikidata.org.crt]:
 Could not evaluate: Could not retrieve information from environment production 
source(s) puppet:///files/ssl/query-preview.wikidata.org.crt

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Ah, so poking around the `certificate.manifests.d` repo I see certs that 
don't necessarily follow the `discovery.wmnet` pattern. To me that implies 
**Option 2** should be working, so I might be missing something. Here's an 
example that doesn't use discovery:
  
ryankemper@puppetmaster1001:/srv/private$ cat 
modules/secret/secrets/certificates/certificate.manifests.d/analytics_http_ui.certs.yaml
yarn.wikimedia.org:
  authority: puppet_ca
  expiry: null
  alt_names: ["yarn.wikimedia.org", "hue.wikimedia.org", 
"hue-next.wikimedia.org", "superset.wikimedia.org", "pivot.wikimedia.org", 
"turnilo.wikimedia.org", "stats.wikimedia.org", "analytics.wikimedia.org", 
"piwik.wikimedia.org", "datasets.wikimedia.org"]
  key:
password: REDACTED
algorithm: ec

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **Option 2 ** fails to even generate the cert. All the cergen documentation 
is written for a certificate like `query-preview.discovery.wmnet` and not 
`wdqs1009.eqiad.wmnet` or `query-preview.wikidata.org`. So I do think this just 
isn't what cergen is built to do.
  
ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Untracked files:
  (use "git add ..." to include in what will be committed)


modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml

nothing added to commit but untracked files present (use "git add" to track)
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' 
--generate --base-path /srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-03-12 08:06:03,210 INFO cergen   
Generating certificates [] with force=False

Status of certificates []

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Hit a big blocker with the current proposed approach of using 
`wdqs1009.eqiad.wmnet` as the cert name:
  
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' 
--generate --base-path /srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-03-12 07:55:03,774 INFO cergen   
Generating certificates ['wdqs1009.eqiad.wmnet'] with force=False
2021-03-12 07:55:03,774 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating all files, force=False...
2021-03-12 07:55:03,776 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating certificate file
/usr/lib/python3/dist-packages/urllib3/connection.py:362: 
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no 
`subjectAltName`, falling back to check for a `commonName` for now. This 
feature is being removed by major browsers and deprecated by RFC 2818. (See 
https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Traceback (most recent call last):
  File "/usr/bin/cergen", line 11, in 
load_entry_point('cergen==0.2.5', 'console_scripts', 'cergen')()
  File "/usr/lib/python3/dist-packages/cergen/main.py", line 93, in main
certificate.generate(force=args['--force'])
  File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 304, in 
generate
self.generate_crt(force=force)
  File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 344, in 
generate_crt
self.x509_cert = self.authority.sign(csr, self.expiry)
  File "/usr/lib/python3/dist-packages/cergen/puppet.py", line 193, in sign
self.name, common_name
RuntimeError: puppetmaster1001.eqiad.wmnet_8140 already has a signed 
certificate for wdqs1009.eqiad.wmnet. If you are trying to regenerate a Puppet 
CA signed certificate, you need to first remove the certificate from the Puppet 
CA. On the puppetmaster, `puppet cert clean wdqs1009.eqiad.wmnet` should do it.
  
  Corresponding to this cergen manifest file 
`modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml`:
  
wdqs1009.eqiad.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: ["wdqs1009.eqiad.wmnet","query-preview.wikidata.org"]
  key:
password: REDACTED
algorithm: ec
  
  I'm a bit lost on the right way to proceed. Thinking out loud here:
  
  **Option 1** Just actually re-generate the cert with `puppet cert clean 
wdqs1009.eqiad.wmnet` before running cergen. This feels wrong though, and I 
think ultimately it comes down to using cergen in a way that wasn't intended: 
name collision with the existing cert puppet has.
  
  **Option 2** Do this instead:
  
query-preview.wikidata.org:
  authority: puppet_ca
  expiry: null
  alt_names: ["query-preview.wikidata.org", "wdqs1009.eqiad.wmnet"]
  key:
password: REDACTED
algorithm: ec
  
  Might want to talk to traffic again. But in the interest of iteration speed I 
think trying out Option **2** is not a bad idea; there's not really any risk 
since this is pertaining to `wdqs1009.eqiad.wmnet` which is a test host.

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment.


  Finished rolling back to the previous iteration of `wdqs.discovery.wmnet` 
cert since we're now going to create a net-new cert `wdqs1009.eqiad.wmnet` for 
`wdqs-test`

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment.


Certificate(wdqs.discovery.wmnet, 
authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.741452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.745452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem:
 PRESENT (mtime: 2021-03-10T19:17:36.206506)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem:
 PRESENT (mtime: 2019-12-10T14:26:42.496697)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12:
 PRESENT (mtime: 2021-03-10T19:17:36.226506)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks:
 PRESENT (mtime: 2021-03-10T19:17:38.030503)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks:
 PRESENT (mtime: 2021-03-10T19:17:38.938502)


ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   
modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-09 Thread RKemper
RKemper added a comment.


Certificate(wdqs.discovery.wmnet, 
authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.741452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.745452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem:
 PRESENT (mtime: 2021-03-10T04:58:24.029629)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem:
 PRESENT (mtime: 2019-12-10T14:26:42.496697)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12:
 PRESENT (mtime: 2021-03-10T04:58:24.049629)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks:
 PRESENT (mtime: 2021-03-10T04:58:25.893621)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks:
 PRESENT (mtime: 2021-03-10T04:58:26.865617)


ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   
modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12

no changes added to commit (use "git add" and/or "git commit -a")
  
  Relevant status after performing the cergen. (Git commit not created yet as 
seen by the dirty working directory)

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T252504: Automate the smoke test of the canary deployment of WDQS

2021-03-08 Thread RKemper
RKemper renamed this task from "Smoke test the canary deployment of WDQS" to 
"Automate the smoke test of the canary deployment of WDQS".
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T252504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Zbyszko, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, 
Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Addshore, Mbch331, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-04 Thread RKemper
RKemper added a comment.


  Here's how the ats mapping looks afterdeploy of the `backend.yaml` changes:
  
ryankemper@cp1075:~$ sudo cat /etc/trafficserver/remap.config | grep query
map http://query-preview.wikidata.org/bigdata 
https://wdqs1009.eqiad.wmnet/bigdata
map http://query-preview.wikidata.org/sparql 
https://wdqs1009.eqiad.wmnet/sparql
map http://query-preview.wikidata.org 
https://webserver-misc-apps.discovery.wmnet
map http://query.wikidata.org/bigdata/ldf 
https://wdqs1005.eqiad.wmnet/bigdata/ldf
map http://query.wikidata.org/sparql https://wdqs.discovery.wmnet/sparql
map http://query.wikidata.org/bigdata https://wdqs.discovery.wmnet/bigdata
map http://query.wikidata.org https://webserver-misc-apps.discovery.wmnet

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-03 Thread RKemper
RKemper added a comment.


  Posting logs of our IRC convo from ~1 month ago for context when I tag people 
for review:
  
[2021-02-04 11:31:36]  bblack: around ?
[2021-02-04 11:31:58]  I'm here with ryankemper to talk about 
T266470, unless you want to jump in a meet
[2021-02-04 11:31:59]  T266470: Expose wdqs1009 to wdqs users and 
gather feedback - https://phabricator.wikimedia.org/T266470
[2021-02-04 11:32:51]  taking notes in 
https://etherpad.wikimedia.org/p/expose-wdqs1009
[2021-02-04 11:33:08]  gehel: hi :)
[2021-02-04 11:33:15]  o/
[2021-02-04 11:33:15]  ryankemper: hi too :)
[2021-02-04 11:33:23]  \o
[2021-02-04 11:33:40]  so, I guess my best understanding of the 
present stuff is what I see in:
[2021-02-04 11:33:47]  context: we want to expose wdqs1009 as a test 
server so that our users can make sure we're not breaking anything with the new 
WDQS updater
[2021-02-04 11:33:53]  
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#157
[2021-02-04 11:34:12]  there's several related definitions there 
for https://query.wikidata.org/
[2021-02-04 11:34:24]  for different subspaces of the URI routing 
to different backends
[2021-02-04 11:34:40]   /bigdata/ldf goes explicitly to wds1005
[2021-02-04 11:35:07]   /bigdata and /sparql both go to the 
discovery service to a set of wdqs
[2021-02-04 11:35:25]  but the root url hits our misc webapps 
setup, for the JS powering the query UI
[2021-02-04 11:35:29]  yep, somewhat similar, but in this case, we 
want to have a different FQDN (probably query-preview.wikidata.org or something 
similar) and route to a different server that isn't part of the current WDQS 
pools
[2021-02-04 11:36:45]  so in this case, I would guess, you'd want 
no special exception for /bigdata/ldf, and you'd want all of /bigdata and 
/sparql -> wdqs1009
[2021-02-04 11:37:06]  which is something we can just define in 
that file I linked, and adding a name at the DNS layer, and done
[2021-02-04 11:37:16]  but the root URL for the UI part, might be a 
little trickier
[2021-02-04 11:37:39]  I need to see where that's defined in git, 
and it might need a new site definition, and to template in a new place to send 
the UI queries...
[2021-02-04 11:37:57]  but there's no other special requirements I 
think/expect?
[2021-02-04 11:37:57]  we'll probably want to create another 
microsite for the JS part, that's fairly straightforward
[2021-02-04 11:38:15]  no other special requirements IIRC
[2021-02-04 11:38:17]  yep, that seems fairly simple to me
[2021-02-04 11:38:22]  so to resume:
[2021-02-04 11:39:08]  create a DNS entry (CNAME to dyna.wm.o), 
another set of entries in backend.yaml map, create another minisite (with the 
appropriate configuration)
[2021-02-04 11:39:22]  do we need to do anything for SSL certs? Or 
is that fully automated?
[2021-02-04 11:39:48]  that part's all shared between everything 
that runs through the cache, so long as you stick to our canonical domains and 
just change the leading hostname part
[2021-02-04 11:40:04]  to e.g. query-preview.wikidata.org
[2021-02-04 11:40:05]  "canonical" includes wikidata.org?
[2021-02-04 11:40:11]  ok, good!
[2021-02-04 11:40:37]  yeah for reference, the canonical set is: 
https://wikitech.wikimedia.org/wiki/HTTPS#For_the_Foundation's_canonical_domain_names
[2021-02-04 11:41:13]  ok, one last stupid and off topic question: 
what does "dyna" stands for in dyna.wm.o?
[2021-02-04 11:42:14]  it's not a "real" DNS RR-type you would see 
in the wild.  It's a custom thing our authserver implements which provides 
dynamic A and  records depending on geodns.  that's how we route to the 
closest of our geographic edges for users
[2021-02-04 11:42:36]  
https://github.com/gdnsd/gdnsd/wiki/GdnsdPluginGeoip
[2021-02-04 11:42:41]  ^ that, basically
[2021-02-04 11:42:57]  thanks!
[2021-02-04 11:43:23]  templates/wikimedia.org:dyna600 
IN DYNA geoip!text-addrs
[2021-02-04 11:43:36]  ^ is what all the other production CNAMEs 
are pointing towards
[2021-02-04 11:43:54]  gehel: also I don't know if you've created 
new service hostnames since the move to netbox
[2021-02-04 11:44:03]  but we don't do commits on the ops/dns repo 
for that kind of thing anymore
[2021-02-04 11:44:25]  Oh no, I havent
[2021-02-04 11:44:33]  ryankemper: is the one who's goign to do it
[2021-02-04 11:45:00]  I assume it's documented
[2021-02-04 11:45:01]  oh wait, I'm getting ahead of ourselves
[2021-02-04 11:45:03] — gehel is searching
[2021-02-04 11:45:13]  that part still is done in ops/dns ! :)
[2021-02-04 11:45:25]  now I'm curious anyway!
[2021-02-04 11:45:43]  
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment.


  Note: Puppet is still disabled on `wdqs2008` while the reload runs. It 
occurred to me that I'm not sure if puppet actually **needs** to be disabled 
during data reloads or if that's just a precaution we've historically taken - 
any insight here @Gehel?

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T274751: Upgrade firmware on wdqs1009

2021-02-25 Thread RKemper
RKemper added a comment.


  @Cmjohnson The data reload is complete on `wdqs1009`, so the host can now 
have its firmware upgraded and be rebooted at its convenience. Note this is an 
internal wdqs test host, so there is no public-facing service for us to worry 
about.
  
  Feel free to proceed when convenient for you - just shoot me a ping or drop a 
line in #wikimedia-discovery, but no need to schedule a formal window.

TASK DETAIL
  https://phabricator.wikimedia.org/T274751

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Cmjohnson, RKemper
Cc: Jclark-ctr, RKemper, Cmjohnson, Gehel, MoritzMuehlenhoff, Ramtin0071, 
MPhamWMF, Devnull, lmata, Muchiri124, CBogen, Akuckartz, Legado_Shulgin, 
Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Southparkfan, faidon, Addshore, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment.


  Downtimed `wdqs2008` until `2021-03-04 21:56:59`

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-08 Thread RKemper
RKemper added a comment.


  Still waiting for the latest dumps to be downloaded (few more hours), then 
need to reboot WDQS hosts as part of https://phabricator.wikimedia.org/T274213, 
then can do the actual data-reload

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-02-08 Thread RKemper
RKemper added a comment.


  (See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket 
has been closed)

TASK DETAIL
  https://phabricator.wikimedia.org/T266495

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper added a comment.


  Updated:
  
  [X] https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates
  [X] 
https://commons.wikimedia.org/wiki/Commons:Village_pump#Unscheduled_maintenance%3A_Wikimedia_Commons_Query_Service
  [X] WikiData ML mailing list (note: A community member pointed out that 
sending an e-mail to a Commons mailing list would have made more sense, which 
is a great point. We'll do that in the future)

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  WCQS is back in service; updating the notification channels right now and 
will comment back here after

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-05 Thread RKemper
RKemper added a comment.


  `sudo cookbook sre.wdqs.data-reload wdqs1009.eqiad.wmnet 
--reuse-downloaded-dump --reload-data wikidata --skolemize --reason 'T267927: 
Reload wikidata jnl from fresh dumps' --task-id T267927 ` is failing with:
  
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -

- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -

- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -

- OUTPUT of 'rm -rf /srv/wdqs...d -- --skolemize' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
22:46:13.025 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error 
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{' 
[line 2]
at 
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFParserHelper.java:440)
at 
org.openrdf.rio.helpers.RDFParserBase.reportFatalError(RDFParserBase.java:685)
at 
org.openrdf.rio.turtle.TurtleParser.reportFatalError(TurtleParser.java:1405)
at org.openrdf.rio.turtle.TurtleParser.parseValue(TurtleParser.java:662)
at 
org.openrdf.rio.turtle.TurtleParser.parsePredicate(TurtleParser.java:505)
at 
org.openrdf.rio.turtle.TurtleParser.parsePredicateObjectList(TurtleParser.java:420)
at 
org.openrdf.rio.turtle.TurtleParser.parseImplicitBlank(TurtleParser.java:613)
at 
org.openrdf.rio.turtle.TurtleParser.parseTriples(TurtleParser.java:394)
at 
org.openrdf.rio.turtle.TurtleParser.parseStatement(TurtleParser.java:259)
at org.openrdf.rio.turtle.TurtleParser.parse(TurtleParser.java:214)
at org.wikidata.query.rdf.tool.Munge.run(Munge.java:105)
at org.wikidata.query.rdf.tool.Munge.main(Munge.java:59)
22:46:13.024 
[org.wikidata.query.rdf.tool.rdf.AsyncRDFHandler$RDFActionsReplayer] INFO  
org.wikidata.query.rdf.tool.Munge - Switching to 
/srv/wdqs/munged/wikidump-1.ttl.gz

  
  `sudo cookbook sre.wdqs.data-reload wdqs1010.eqiad.wmnet 
--reuse-downloaded-dump --reload-data wikidata --reason 'T267927: Reload 
wikidata jnl from fresh dumps' --task-id T267927`is failing with:
  
ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-reload 
wdqs1010.eqiad.wmnet --reuse-downloaded-dump --reload-data wikidata --reason 
'T267927: Reload wikidata jnl from fresh dumps' --task-id T267927
START - Cookbook sre.wdqs.data-reload
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -

PASS 
||
 100% (1/1) [00:00<00:00,  1.92hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 'test -f 
/srv/wdq...test-all.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Detected dump (/srv/wdqs/latest-all.ttl.bz2). Skipping download
- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -

PASS 
||
 100% (1/1) [00:00<00:00,  2.44hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 'test -f 
/srv/wdq...-lexemes.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Detected dump (/srv/wdqs/latest-lexemes.ttl.bz2). Skipping download
checking available disk space
- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -

PASS 
||
 100% (1/1) [00:00<00:00,  2.45hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 
'dump_size=`du /s...avail+$db_size))'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Running munger for main database and then lexeme
munging /srv/wdqs/munged (skolemizaton: False)
- OUTPUT of 'rm -rf /srv/wdqs.../wdqs/munged -- ' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
22:50:03.137 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error 
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{' 
[line 2]
at 
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFPar

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-02-04 Thread RKemper
RKemper added a comment.


  TODO from IRC meeting with bblack/gehel: create a DNS entry (CNAME to 
dyna.wm.o), another set of entries in backend.yaml map, create another minisite 
(with the appropriate configuration)

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


  1   2   >