[Wikidata-bugs] [Maniphest] T301461: Investigate cache issues after WDQS UI deployments
RKemper added a comment. In T301461#8014754 <https://phabricator.wikimedia.org/T301461#8014754>, @Lucas_Werkmeister_WMDE wrote: > @RKemper thanks, that would be great! Sorry for the delay here, we had some confusion as to whether the correct file was modified in the previous patch (spoiler alert: it was!). Just reloaded the config now and tests are now passing after a quick patch <https://gerrit.wikimedia.org/r/c/operations/puppet/+/807200>: ryankemper@cumin1001:~$ httpbb /srv/deployment/httpbb-tests/query_service/test_wdqs.yaml --hosts wdqs1012.eqiad.wmnet Sending to wdqs1012.eqiad.wmnet... PASS: 4 requests sent to wdqs1012.eqiad.wmnet. All assertions passed. We should be good here, although I'll defer to @Lucas_Werkmeister_WMDE for any further testing TASK DETAIL https://phabricator.wikimedia.org/T301461 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, EBernhardson, RLazarus, ItamarWMDE, toan, Lucas_Werkmeister_WMDE, Aklapper, Fernandobacasegua34, Astuthiodit_1, AWesterinen, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, EBjune, merbst, LawExplorer, Salgo60, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301461: Investigate cache issues after WDQS UI deployments
RKemper added a comment. In T301461#7994256 <https://phabricator.wikimedia.org/T301461#7994256>, @Lucas_Werkmeister_WMDE wrote: > Status update: the Puppet change has been merged, but apparently the Apache config isn’t reloaded automatically. @RLazarus indicated on Gerrit that @EBernhardson would be the right person for that. @Lucas_Werkmeister_WMDE thanks for the heads up, I'll get the configs reloaded on Monday. TASK DETAIL https://phabricator.wikimedia.org/T301461 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, EBernhardson, RLazarus, ItamarWMDE, toan, Lucas_Werkmeister_WMDE, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, Mahir256, QZanden, EBjune, merbst, LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
RKemper added a comment. @Dominicbm We made `the error message is understandable by users` part of the acceptance criteria of this ticket, so I think that's why T306919 <https://phabricator.wikimedia.org/T306919> was closed as a duplicate. Basically this ticket covers both the fixing of the actual problem as well as seeing what can be done to make the error message itself more comprehensible. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, RKemper Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
RKemper added a comment. @Dominicbm @Fuzheado Understood that you all are still seeing these failures. We're still having trouble reproducing it on our end (we don't have great intuition as why but it's possible that the issue might only surface based on the geoDNS routing). We've added some further logging; next time any of you run into this issue if you could drop by the wikimedia-search and let us know, we can take a look at the logs and see if we can glean anything. Meanwhile on our end we'll keep trying to hone in on the source of the failure. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, RKemper Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead
RKemper removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, RKemper Cc: RKemper, bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Fernandobacasegua34, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan, Neuronton ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead
RKemper reopened this task as "In Progress". RKemper added a comment. Oops, I just meant to move on workboard, not sure how I closed it as well TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, RKemper Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead
RKemper closed this task as "Resolved". RKemper moved this task from Needs review to Waiting on the Discovery-Search (Current work) board. RKemper added a comment. Moving to `Waiting` while we see how the newest settings do TASK DETAIL https://phabricator.wikimedia.org/T293862 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, RKemper Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead
RKemper added a comment. To check for presence of touched file: ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-public' '[ -f "/tmp/wdqs_blazegraph_jvmquake_warn_gc" ] && echo yes || echo no' 11 hosts will be targeted: wdqs[2001-2004,2007].codfw.wmnet,wdqs[1004-1007,1012-1013].eqiad.wmnet Ok to proceed on 11 hosts? Enter the number of affected hosts to confirm or "q" to quit 11 = NODE GROUP = (5) wdqs[2001-2002,2007].codfw.wmnet,wdqs[1004-1005].eqiad.wmnet - OUTPUT of '[ -f "/tmp/wdqs_...o yes || echo no' - no = NODE GROUP = (6) wdqs[2003-2004].codfw.wmnet,wdqs[1006-1007,1012-1013].eqiad.wmnet - OUTPUT of '[ -f "/tmp/wdqs_...o yes || echo no' - yes PASS |███| 100% (11/11) [00:00<00:00, 12.38hosts/s] FAIL || 0% (0/11) [00:00= 100.0% threshold) for command: '[ -f "/tmp/wdqs_...o yes || echo no'. 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. To clear away current touched files: ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-public' "rm -fv '/tmp/wdqs_blazegraph_jvmquake_warn_gc'" 11 hosts will be targeted: wdqs[2001-2004,2007].codfw.wmnet,wdqs[1004-1007,1012-1013].eqiad.wmnet Ok to proceed on 11 hosts? Enter the number of affected hosts to confirm or "q" to quit 11 = NODE GROUP = (6) wdqs[2003-2004].codfw.wmnet,wdqs[1006-1007,1012-1013].eqiad.wmnet - OUTPUT of 'rm -fv '/tmp/wdq...vmquake_warn_gc'' - removed '/tmp/wdqs_blazegraph_jvmquake_warn_gc' PASS |███| 100% (11/11) [00:00<00:00, 12.55hosts/s] FAIL || 0% (0/11) [00:00= 100.0% threshold) for command: 'rm -fv '/tmp/wdq...vmquake_warn_gc''. 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, RKemper Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad
RKemper moved this task from Waiting to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. Alert appears to have resolved after T301108 <https://phabricator.wikimedia.org/T301108> F35007412: image.png <https://phabricator.wikimedia.org/F35007412> TASK DETAIL https://phabricator.wikimedia.org/T302830 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Aklapper, RKemper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301650: WCQS "Application Connection Error" E009
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. F34997527: image.png <https://phabricator.wikimedia.org/F34997527> New `cache-control` header is visible in Chrome and Firefox now. We're hopeful that this should fix the problem, but if not we won't spend too much more time digging in. @Dominicbm Let us know if you still experience this behavior in the future. Thanks! TASK DETAIL https://phabricator.wikimedia.org/T301650 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, MPhamWMF, EBernhardson, Zbyszko, Aklapper, Dominicbm, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, FRomeo_WMF, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Nintendofan885, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, JKSTNK, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T302830 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T302830 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad
RKemper triaged this task as "High" priority. RKemper added a project: Wikidata-Query-Service. TASK DETAIL https://phabricator.wikimedia.org/T302830 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.
RKemper added a project: SRE-OnFire. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RKemper Cc: Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, Addshore, Aklapper, Fernandobacasegua34, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, LSobanski, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, jijiki, Klaas_Z4us_V, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, elukey, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301953: Investigate wdqs1013 stability issues
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T301953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Zbyszko, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301167: Service implementation for wdqs20[09,10,11,12]
RKemper created this task. RKemper added projects: Discovery-Search, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Procurement by dc-ops done in T291982; racking by dc-ops done in T294297 <https://phabricator.wikimedia.org/T294297> AC [ ] 4 new hosts (3 refresh, 1 expansion), `wdqs20[09,10,11,12]` brought into service [ ] wdqs200[1-3] decom'd after new hosts brought into service Note - https://docs.google.com/spreadsheets/d/1EYK0x4GCxv1fG77Co-S-uBiPqrvXRiLzIW34kOAJWws/edit#gid=342761434 shows wdqs2001 and wdqs2002 should be decom'd ASAP to reduce power draw in the DCs. So this ticket will take care of that (given that decom'ing the old hosts is part of the AC) TASK DETAIL https://phabricator.wikimedia.org/T301167 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, RKemper, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T301147 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Michael, Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T258754: Define SLOs and error budget for WDQS
RKemper added a comment. Ryan will add a quick blurb in https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service about the SLO with a link to the dashboard, at which point we can move this to `Needs Reporting`. Also we (Ryan/Brian) should mention the SLO in our next SRE meeting. TASK DETAIL https://phabricator.wikimedia.org/T258754 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, Addshore, Gehel, RKemper, Aklapper, bking, Invadibot, maantietaja, CBogen, Akuckartz, ET4Eva, Nandana, Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T258754: Define SLOs and error budget for WDQS
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T258754 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, Addshore, Gehel, RKemper, Aklapper, bking, Invadibot, maantietaja, CBogen, Akuckartz, ET4Eva, Nandana, Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T264181: Migrate WDQS to profile::java
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search Platform team. Restricted Application added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T264181 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, MoritzMuehlenhoff, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T193473: Add HTTPS support to wdqs-internal service
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search Platform team. TASK DETAIL https://phabricator.wikimedia.org/T193473 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Smalyshev, Gehel, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, ET4Eva, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T270614: Automatically depool wdqs servers that are "lagged"
RKemper added a parent task: T297555: [epic] Brian's onboarding to the Search Platform team. TASK DETAIL https://phabricator.wikimedia.org/T270614 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: So9q, Addshore, Moebeus, Nikki, Mahir256, Lydia_Pintscher, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T193473: Add HTTPS support to wdqs-internal service
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T193473 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Smalyshev, Gehel, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, CBogen, Akuckartz, RhinosF1, ET4Eva, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances
RKemper added a comment. In T294961#7489817 <https://phabricator.wikimedia.org/T294961#7489817>, @Gehel wrote: > Let's make sure the kernel version is pinned somewhere in our puppet code! Then we can wait to see if the problem is reproduced or not. > ryankemper, ebernhardson: yeah, let's install the kernel package manually for testing and if it fixes the issue, then we can apply the profile::base::linux510 So specifically we'll apply `profile::base::linux510` TASK DETAIL https://phabricator.wikimedia.org/T294961 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances
RKemper merged a task: T294865: wcqs1002 and wcqs2001 unresponsive. RKemper added a subscriber: Dzahn. TASK DETAIL https://phabricator.wikimedia.org/T294961 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive
RKemper closed this task as a duplicate of T294961: Resolve kernel hang on wcqs* instances. TASK DETAIL https://phabricator.wikimedia.org/T294865 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Dzahn, RKemper, Gehel, Aklapper, 786, Suran38, Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances
RKemper added a comment.
DNS changes from https://gerrit.wikimedia.org/r/c/operations/dns/+/736585
-
ryankemper@authdns1001:~$ sudo -i authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 2c9042d86630ef640aaf1ab101c9f98bda1824e0...
templates/wmnet| 1 -
utils/mock_etc/discovery-geo-resources | 1 -
2 files changed, 2 deletions(-)
diff --git templates/wmnet templates/wmnet
index c89c1610..041fc7b9 100644
--- templates/wmnet
+++ templates/wmnet
@@ -654,7 +654,6 @@ push-notifications 300/10 IN DYNA
geoip!disc-push-notifications
recommendation-api 300/10 IN DYNA geoip!disc-recommendation-api
wdqs 300/10 IN DYNA geoip!disc-wdqs
wdqs-internal 300/10 IN DYNA geoip!disc-wdqs-internal
-wcqs 300/10 IN DYNA geoip!disc-wcqs
zotero 300/10 IN DYNA geoip!disc-zotero
blubberoid 300/10 IN DYNA geoip!disc-blubberoid
eventgate-analytics 300/10 IN DYNA geoip!disc-eventgate-analytics
diff --git utils/mock_etc/discovery-geo-resources
utils/mock_etc/discovery-geo-resources
index 64e612b3..41a50fc4 100644
--- utils/mock_etc/discovery-geo-resources
+++ utils/mock_etc/discovery-geo-resources
@@ -66,4 +66,3 @@ disc-shellbox-media => { map => mock, dcmap => {
mock => 192.0.2.1 } }
disc-shellbox-syntaxhighlight => { map => mock, dcmap => { mock =>
192.0.2.1 } }
disc-shellbox-timeline => { map => mock, dcmap => { mock => 192.0.2.1 } }
disc-tegola-vector-tiles => { map => mock, dcmap => { mock => 192.0.2.1 } }
-disc-wcqs=> { map => mock, dcmap => { mock => 192.0.2.1 } }
Merge these changes? (yes/no)? yes
Updating 7a86417b..2c9042d8
Fast-forward
templates/wmnet| 1 -
utils/mock_etc/discovery-geo-resources | 1 -
2 files changed, 2 deletions(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.oenx7otf
-- Generating zonefiles from zone templates
-- Processed 211 zones into directory /tmp/dns-check.oenx7otf/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 37
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 65
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 132 Warnings, 0 Ignored violations, 0 Ignored lines
-- Copying automatically generated zone files under target tree
-- Copying repo-driven real config files and admin_state
-- Copying puppetized config and GeoIP from /etc/gdnsd
-- Checking for illegal tabs in zonefiles
-- Running zone_validator to check WMF rules
-- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.oenx7otf
-- Preflight checkconf is OK
Deploying from /tmp/dns-check.oenx7otf to system dirs
-- Zone changed: wmnet
-- Descending to subdirectory: netbox
-- Done with subdir: netbox
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---
authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
(11)
---
OK - authdns updated successfully
OK - authdns-update successful on all nodes!
TASK DETAIL
https://phabricator.wikimedia.org/T294961
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache,
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun,
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264,
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301,
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana,
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi,
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0,
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen,
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs,
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g,
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive
RKemper added a subtask: T294961: Resolve kernel hang on wcqs* instances. TASK DETAIL https://phabricator.wikimedia.org/T294865 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Suran38, Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances
RKemper added a parent task: T294865: wcqs1002 and wcqs2001 unresponsive. TASK DETAIL https://phabricator.wikimedia.org/T294961 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T260568: [EPIC] Productionize WCQS
RKemper added a subtask: T294865: wcqs1002 and wcqs2001 unresponsive. TASK DETAIL https://phabricator.wikimedia.org/T260568 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: GFontenelle_WMF, So9q, Fuzheado, Zache, ttaylor, CBogen, Gehel, Aklapper, Suran38, Invadibot, MPhamWMF, maantietaja, Peteosx1x, NavinRizwi, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294865: wcqs1002 and wcqs2001 unresponsive
RKemper added a parent task: T260568: [EPIC] Productionize WCQS. TASK DETAIL https://phabricator.wikimedia.org/T294865 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Suran38, Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T288231 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: elukey, Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293638: Create and update the process for separate WCQS mutation topic
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T293638 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Ottomata, Aklapper, Zbyszko, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T288231 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T288231 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T288231 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, RKemper, Aklapper, dcausse, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288231: Deploy the wdqs streaming updater to production
RKemper added a comment.
[data-transfer cookbook -> kafka offset output]
---
(including this output since it's the first time we've relied on the new
kafka offset part of the data-transfer process; won't be posting the output for
any subsequent cookbooks runs)
Also note that the output is a little tricky to parse. I can't tell if it did
succeed nad the `Error sending OffsetCommitRequest_v2...` log lines were errors
that were resolved upon retry, or if the cookbook falsely reported the offset.
But judging by the fact that `wdqs2005` is reporting the correct number of
triples, I have to assume that it worked properly. (This also lines up with the
fact that we do have log lines that seem to report success: `Offsets set for
target cluster "main", site "codfw" and consumer group "wdqs2005"`)
Transferring Kafka offsets
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0)
IPv6]
Probing node bootstrap-2 broker version
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests
on startup
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:38', 9093, 0, 0)
IPv6]
Probing node bootstrap-1 broker version
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
Broker version identifed as 1.0.0
Set configuration api_version=(1, 0, 0) to skip auto check_version requests
on startup
Same cluster, setting offsets...
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: connecting to
kafka-main2004.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:38', 9093, 0, 0) IPv6]
[IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: connecting to
kafka-main2001.codfw.wmnet:9093 [('2620:0:860:101:10:192:0:17', 9093, 0, 0)
IPv6]
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Connection
complete.
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to
kafka-main2005.codfw.wmnet:9093
[('2620:0:860:104:10:192:48:46', 9093, 0, 0) IPv6]
Group coordinator for wdqs2008 is BrokerMetadata(nodeId='coordinator-2003',
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2008
[IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Connection
complete.
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0)
IPv6]
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection
complete.
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
Extracted offsets from source cluster "main", site "codfw" and consumer
group "wdqs2008".
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: connecting to
kafka-main2005.codfw.wmnet:9093 [('2620:0:860:104:10:192:48:46', 9093, 0, 0)
IPv6]
Group coordinator for wdqs2005 is BrokerMetadata(nodeId='coordinator-2003',
host='kafka-main2003.codfw.wmnet', port=9093, rack=None)
Discovered coordinator coordinator-2003 for group wdqs2005
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: connecting to
kafka-main2003.codfw.wmnet:9093 [('2620:0:860:103:10:192:32:136', 9093, 0, 0)
IPv6]
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Connection
complete.
Error sending OffsetCommitRequest_v2 to node coordinator-2003
[NodeNotReadyError: coordinator-2003]
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Connection complete.
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing
connection.
Offsets set for target cluster "main", site "codfw" and consumer group
"wdqs2005".
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
[IPv6 ('2620:0:860:104:10:192:48:38', 9093, 0, 0)]>: Closing
connection.
[IPv6 ('2620:0:860:101:10:192:0:17', 9093, 0, 0)]>: Closing
connection.
[IPv6 ('2620:0:860:104:10:192:48:46', 9093, 0, 0)]>: Closing
connection.
[IPv6
('2620:0:860:103:10:192:32:136', 9093, 0, 0)]>: Closing connection.
[Wikidata-bugs] [Maniphest] T282117: WCQS needs to be exposed through a wikimedia.org domain
RKemper added a comment.
[Log of failed authdns-update following https://gerrit.wikimedia.org/r/724520]
--
ryankemper@authdns1001:~$ sudo -i authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing c81f47118e6e6fea77cc8970885dd61b61c92bd0...
templates/wmnet| 1 +
utils/mock_etc/discovery-geo-resources | 1 +
2 files changed, 2 insertions(+)
diff --git templates/wmnet templates/wmnet
index ea3d7279..9d554061 100644
--- templates/wmnet
+++ templates/wmnet
@@ -601,6 +601,7 @@ push-notifications 300/10 IN DYNA
geoip!disc-push-notifications
recommendation-api 300/10 IN DYNA geoip!disc-recommendation-api
wdqs 300/10 IN DYNA geoip!disc-wdqs
wdqs-internal 300/10 IN DYNA geoip!disc-wdqs-internal
+wcqs 300/10 IN DYNA geoip!disc-wcqs
zotero 300/10 IN DYNA geoip!disc-zotero
blubberoid 300/10 IN DYNA geoip!disc-blubberoid
eventgate-analytics 300/10 IN DYNA geoip!disc-eventgate-analytics
diff --git utils/mock_etc/discovery-geo-resources
utils/mock_etc/discovery-geo-resources
index d5f686e1..0d51b0bc 100644
--- utils/mock_etc/discovery-geo-resources
+++ utils/mock_etc/discovery-geo-resources
@@ -65,3 +65,4 @@ disc-shellbox-media => { map => mock, dcmap => {
mock => 192.0.2.1 } }
disc-shellbox-syntaxhighlight => { map => mock, dcmap => { mock =>
192.0.2.1 } }
disc-shellbox-timeline => { map => mock, dcmap => { mock => 192.0.2.1 } }
disc-tegola-vector-tiles => { map => mock, dcmap => { mock => 192.0.2.1 } }
+disc-wcqs=> { map => mock, dcmap => { mock => 192.0.2.1 } }
Merge these changes? (yes/no)? yes
Updating e1e708d9..c81f4711
Fast-forward
templates/wmnet| 1 +
utils/mock_etc/discovery-geo-resources | 1 +
2 files changed, 2 insertions(+)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.uhizg7cf
-- Generating zonefiles from zone templates
-- Processed 211 zones into directory /tmp/dns-check.uhizg7cf/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 37
W002|MISSING_PTR_FOR_NAME_AND_IP: 26
W103|MISSING_MGMT_FOR_NAME: 64
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 130 Warnings, 0 Ignored violations, 0 Ignored lines
-- Copying automatically generated zone files under target tree
-- Copying repo-driven real config files and admin_state
-- Copying puppetized config and GeoIP from /etc/gdnsd
-- Checking for illegal tabs in zonefiles
-- Running zone_validator to check WMF rules
-- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.uhizg7cf
Traceback (most recent call last):
File "utils/deploy-check.py", line 276, in
main()
File "utils/deploy-check.py", line 268, in main
deploy_check(args.deploy, args.skip_reload, args.no_gdnsd, Path(tdir),
gdir)
File "utils/deploy-check.py", line 214, in deploy_check
safe_cmd([GDNSD_BIN, '-c', str(tdir), 'checkconf'])
File "utils/deploy-check.py", line 88, in safe_cmd
p_err.decode('utf-8')))
Exception: Command /usr/sbin/gdnsd -c /tmp/dns-check.uhizg7cf checkconf
failed with exit code 42, stderr:
info: gdnsd version 3.8.0 @ pid 22926
info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.154.238:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 208.80.153.231:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 91.198.174.239:53
info: DNS listener threads (8 UDP + 8 TCP) configured for 198.35.27.27:53
info: DNS listener threads (8 TCP PROXY) configured for 127.0.0.1:535
info: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:5353
info: DNS listener threads (1 UDP + 1 TCP) configured for [::]:5353
info: plugin_geoip: map 'generic-map': Loading GeoIP2 database
'/tmp/dns-check.uhizg7cf/geoip/GeoIP2-City.mmdb': Version: 2.0, Type:
GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-27 17:10:01 UTC
info: plugin_geoip: map 'generic-map' runtime db updated. nets: 1122136
dclists: 8
info: plugin_geoip: map 'checkdoh-map' runtime db updated. nets: 410
dclists: 2
info: plugin_geoip: map 'discovery-map': Loading GeoIP2 database
'/tmp/dns-check.uhizg7cf/geoip/GeoIP2-City.mmdb': Version: 2.0, Type:
GeoIP2-City, IPVersion: 6, Timestamp: 2021-09-27 17:10:01 UTC
info: plugin_geoip: map 'discovery-map' runtime db updated. nets: 464
dclists: 2
info: admin_state: checking state file
'/tmp/dns-check.uhizg7cf/state/admin_state'...
error: plugin_geoip: In
[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment
RKemper added a comment. ~/wmf/rdf/queries [master!]% ./test.sh -s http://localhost:/ Running 1592.sparqlcurl -G --data-urlencode query@1592.sparql --data format=json --data random=62170 http://localhost://sparql -s PASSED $KEEP is no, nuked /tmp/querytest.1592.62170 Running 1785.sparqlcurl -G --data-urlencode query@1785.sparql --data format=json --data random=62170 http://localhost://sparql -s PASSED $KEEP is no, nuked /tmp/querytest.1785.62170 Running badservice.sparqlcurl -G --data-urlencode query@badservice.sparql --data format=json --data random=62170 http://localhost://sparql -s FAILED $KEEP is no, nuked /tmp/querytest.badservice.62170 Running cats.sparqlcurl -G --data-urlencode query@cats.sparql --data format=json --data random=62170 http://localhost://sparql -s PASSED $KEEP is no, nuked /tmp/querytest.cats.62170 Running dates.sparqlcurl -G --data-urlencode query@dates.sparql --data format=json --data random=62170 http://localhost://sparql -s PASSED $KEEP is no, nuked /tmp/querytest.dates.62170 Running minus.sparqlcurl -G --data-urlencode query@minus.sparql --data format=json --data random=62170 http://localhost://sparql -s Test queries passed during deploy of https://gerrit.wikimedia.org/r/c/operations/puppet/+/720801 to canary `wdqs1003` TASK DETAIL https://phabricator.wikimedia.org/T280247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, RKemper Cc: WMDE-leszek, RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, ttaylor, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment.
[Steps for deploy of https://gerrit.wikimedia.org/r/723254]
---
merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/723254
(on cumin) => sudo cumin 'O:lvs::balancer' 'sudo run-puppet-agent'
ack alerts
!log T280001 Restarting pybal on backup low-traffic hosts `lvs2010` and
`lvs1016`
sudo cumin 'P{lvs2010*,lvs1016*}' 'sudo systemctl restart pybal'
Sanity check of `sudo ipvsadm -L -n` on low-traffic backups `lvs2010` and
`lvs1016` => contains wcqs and has sane list of backends (`wcqs*`)
!log T280001 Sanity check of `sudo ipvsadm -L -n` on backup `lvs2010` and
`lvs1016` looks good, proceeding
Wait 120s while checking https://icinga.wikimedia.org/alerts
!log Waited 120s and checked https://icinga.wikimedia.org/alerts,
proceeding to primary low-traffic hosts `lvs2009` and `lvs1015`
!log T280001 Restarting pybal on low-traffic primaries `lvs2009` and
`lvs1015`
sudo cumin 'P{lvs2009*,lvs1015*}' 'sudo systemctl restart pybal'
Run a test like `curl -v -k http://wcqs.svc.eqiad.wmnet`
!log T280001 Sanity check of `wcqs.svc.{eqiad,codfw}.wmnet` looks good;
declaring this a success
TASK DETAIL
https://phabricator.wikimedia.org/T280001
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69,
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY,
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314,
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas,
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment. [conftool changes following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959] - Now running conftool-merge to sync any changes to conftool data Running conftool-sync on /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity node from /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/node/codfw.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/node/eqsin.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/node/eqiad.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/node/esams.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/node/ulsfo.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity discovery from /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/discovery/services.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/discovery/mediawiki.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity mwconfig from /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/mwconfig/data.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity dbconfig-instance from /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/dbconfig-instance/instances.yaml 2021-09-23 16:38:10 [INFO] conftool::load_files: Loading data for entity dbconfig-section from /etc/conftool/data 2021-09-23 16:38:10 [INFO] conftool::load_files: Parsing file /etc/conftool/data/dbconfig-section/sections.yaml 2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for node 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags codfw/wcqs/wcqs/wcqs2003.codfw.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags codfw/wcqs/wcqs/wcqs2002.codfw.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags eqiad/wcqs/wcqs/wcqs1001.eqiad.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags eqiad/wcqs/wcqs/wcqs1003.eqiad.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags eqiad/wcqs/wcqs/wcqs1002.eqiad.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Creating node with tags codfw/wcqs/wcqs/wcqs2001.codfw.wmnet 2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for discovery 2021-09-23 16:38:10 [INFO] conftool::load: Creating discovery with tags wcqs/eqiad 2021-09-23 16:38:10 [INFO] conftool::load: Creating discovery with tags wcqs/codfw 2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for mwconfig 2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for dbconfig-instance 2021-09-23 16:38:10 [INFO] conftool::load: Adding objects for dbconfig-section 2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for dbconfig-section 2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for dbconfig-instance 2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for mwconfig 2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for discovery 2021-09-23 16:38:10 [INFO] conftool::load: Removing stale objects for node TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment.
[DNS change -> sre.dns.netbox cookbook]
---
ryankemper@cumin1001:~$ sudo -i cookbook sre.dns.netbox -t T280001 'Added
wcqs.svc.[eqiad,codfw].wmnet'
START - Cookbook sre.dns.netbox
Generating the DNS records from Netbox data. It will take a couple of
minutes.
- OUTPUT of 'cd /tmp && runus...ad,codfw].wmnet"' -
2021-09-21 18:57:04,823 [INFO] Gathering devices, interfaces, addresses and
prefixes from Netbox
2021-09-21 19:00:22,214 [WARNING] Device frqueue1002 of IP 10.64.40.204/26
not in devices, skipping.
2021-09-21 19:00:22,219 [WARNING] Device phab1003 of IP 10.65.1.16/16 not
in devices, skipping.
2021-09-21 19:00:22,262 [INFO] Gathered 2307 devices from Netbox
2021-09-21 19:00:22,262 [INFO] Generating DNS records
2021-09-21 19:00:32,939 [INFO] Generated 13348 direct and reverse records
(6674 each) in 28 direct zones and 171 reverse zones
2021-09-21 19:00:32,940 [INFO] Cloning /srv/netbox-exports/dns.git/ to
/tmp/dns-c25pcHBldHM-5qrdksyv ...
2021-09-21 19:00:33,215 [INFO] Generating zonefile snippets to directory
/tmp/dns-c25pcHBldHM-5qrdksyv
2021-09-21 19:00:34,249 [INFO] Committed changes:
4a37a231dbab6bfb2f424feffef4d4f411b8dd49
2021-09-21 19:00:34,285 [INFO] Validating generated data
2021-09-21 19:00:34,286 [INFO] Commit details: {'insertions': 4,
'deletions': 0, 'lines': 4, 'files': 4}
commit 4a37a231dbab6bfb2f424feffef4d4f411b8dd49
Author: generate-dns-snippets
Date: Tue Sep 21 19:00:34 2021 +
ryankemper@cumin1001: Added wcqs.svc.[eqiad,codfw].wmnet
diff --git a/1.2.10.in-addr.arpa b/1.2.10.in-addr.arpa
index 16f68f5..0d9297a 100644
--- a/1.2.10.in-addr.arpa
+++ b/1.2.10.in-addr.arpa
@@ -53,3 +53,4 @@
64 1H IN PTR shellbox-media.svc.codfw.wmnet.
65 1H IN PTR shellbox-syntaxhighlight.svc.codfw.wmnet.
66 1H IN PTR shellbox-timeline.svc.codfw.wmnet.
+67 1H IN PTR wcqs.svc.codfw.wmnet.
diff --git a/2.2.10.in-addr.arpa b/2.2.10.in-addr.arpa
index adeb0c3..6904f40 100644
--- a/2.2.10.in-addr.arpa
+++ b/2.2.10.in-addr.arpa
@@ -58,3 +58,4 @@
64 1H IN PTR shellbox-media.svc.eqiad.wmnet.
65 1H IN PTR shellbox-syntaxhighlight.svc.eqiad.wmnet.
66 1H IN PTR shellbox-timeline.svc.eqiad.wmnet.
+67 1H IN PTR wcqs.svc.eqiad.wmnet.
diff --git a/svc.codfw.wmnet b/svc.codfw.wmnet
index 23001c2..965466b 100644
--- a/svc.codfw.wmnet
+++ b/svc.codfw.wmnet
@@ -50,6 +50,7 @@ thanos-swift 1H IN A 10.2.1.54
thumbor 1H IN A 10.2.1.24
toolhub 1H IN A 10.2.1.62
videoscaler 1H IN A 10.2.1.5
+wcqs 1H IN A 10.2.1.67
wdqs 1H IN A 10.2.1.32
wdqs-internal1H IN A 10.2.1.41
wikifeeds1H IN A 10.2.1.47
diff --git a/svc.eqiad.wmnet b/svc.eqiad.wmnet
index 3dd0aa1..185c684 100644
--- a/svc.eqiad.wmnet
+++ b/svc.eqiad.wmnet
@@ -57,6 +57,7 @@ thanos-swift 1H IN A 10.2.2.54
thumbor 1H IN A 10.2.2.24
toolhub 1H IN A 10.2.2.62
videoscaler 1H IN A 10.2.2.5
+wcqs 1H IN A 10.2.2.67
wdqs 1H IN A 10.2.2.32
wdqs-internal1H IN A 10.2.2.41
wikifeeds1H IN A 10.2.2.47
METADATA: {"path": "/tmp/dns-c25pcHBldHM-5qrdksyv", "sha1":
"4a37a231dbab6bfb2f424feffef4d4f411b8dd49", "insertions": 4, "deletions": 0,
"lines": 4, "files": 4}
PASS
|█|
100% (1/1) [03:30<00:00, 210.69s/hosts]
FAIL |
| 0% (0/1) [03:30= 100.0% threshold) for command: 'cd /tmp &&
runus...ad,codfw].wmnet"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully
executed all commands.
>>> Have you checked that the diff is OK?
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment.
[DNS change -> eqiad validation]
ryankemper@authdns1001:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t
any wcqs.svc.eqiad.wmnet ; done
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns0.wikimedia.org -t any
wcqs.svc.eqiad.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48639
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 2488f3a7fd0c5bbe01d8534adfbac196 (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.eqiad.wmnet. IN ANY
;; ANSWER SECTION:
wcqs.svc.eqiad.wmnet. 3600IN HINFO "RFC8482" ""
;; Query time: 0 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Tue Sep 21 18:49:13 UTC 2021
;; MSG SIZE rcvd: 96
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns1.wikimedia.org -t any
wcqs.svc.eqiad.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43542
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 4ea9c8d14a719114988252f4b7a24024 (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.eqiad.wmnet. IN ANY
;; ANSWER SECTION:
wcqs.svc.eqiad.wmnet. 3600IN HINFO "RFC8482" ""
;; Query time: 0 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Tue Sep 21 18:49:13 UTC 2021
;; MSG SIZE rcvd: 96
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns2.wikimedia.org -t any
wcqs.svc.eqiad.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1138
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 0e7f90b6a29651b34d8115b3a817a4cd (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.eqiad.wmnet. IN ANY
;; ANSWER SECTION:
wcqs.svc.eqiad.wmnet. 3600IN HINFO "RFC8482" ""
;; Query time: 0 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Tue Sep 21 18:49:13 UTC 2021
;; MSG SIZE rcvd: 96
[DNS change -> codfw validation]
ryankemper@authdns1001:~$ for i in 0 1 2 ; do dig @ns${i}.wikimedia.org -t
any wcqs.svc.codfw.wmnet ; done
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns0.wikimedia.org -t any
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36183
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 8dad192df87e3fd3a2874c513054386d (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.codfw.wmnet. IN ANY
;; ANSWER SECTION:
wcqs.svc.codfw.wmnet. 3600IN HINFO "RFC8482" ""
;; Query time: 0 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Tue Sep 21 18:49:39 UTC 2021
;; MSG SIZE rcvd: 96
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns1.wikimedia.org -t any
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10558
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
; COOKIE: 8b49e3f1d8cc32489ef6822def8f7e99 (good)
; OPT=11: 01 72 (".r")
;; QUESTION SECTION:
;wcqs.svc.codfw.wmnet. IN ANY
;; ANSWER SECTION:
wcqs.svc.codfw.wmnet. 3600IN HINFO "RFC8482" ""
;; Query time: 0 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Tue Sep 21 18:49:39 UTC 2021
;; MSG SIZE rcvd: 96
; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @ns2.wikimedia.org -t any
wcqs.svc.codfw.wmnet
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- o
[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw
RKemper added a comment. We're no longer doing hourly restarts; we should monitor availability over the weekend to make sure that we're capable of handling current load without the crutch of the hourly restarts. We still have the useragent we banned in the last full outage (dailymotion) banned, so I'd think we'll be okay on load, but I'll be checking Grafana over the next couple days. We can/should unban dailymotion early next week - perhaps tuesday - if wdqs has been stable over the weekend and during monday's load TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Ladsgroup, Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment
RKemper added a comment. @Ladsgroup https://gerrit.wikimedia.org/r/c/operations/puppet/+/719502 Okay the puppet change is officially merged, so the `gui-deploy` stuff should be unblocked now. TASK DETAIL https://phabricator.wikimedia.org/T280247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, RKemper Cc: WMDE-leszek, RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, ttaylor, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290545: Wikidata Query Service inaccessible via GUI, only partially accessible via embedded queries
RKemper added a comment. Config change was successfully reverted. We had to clear out the varnish cache with `echo 'https://query.wikidata.org/' | mwscript purgeList.php` per https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging. Everything should be / is back to normal now. TASK DETAIL https://phabricator.wikimedia.org/T290545 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lydia_Pintscher, Ladsgroup, RKemper, Gnoeee, Fuzheado, Aklapper, Daniel_Mietchen, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290545: Wikidata Query Service inaccessible via GUI, only partially accessible via embedded queries
RKemper added a comment. @Lydia_Pintscher Thanks this is definitely related to that work. When I spot-checked query.wikidata.org after merging earlier it looks like the new config hadn't taken effect yet. Reverted the puppet change. Working on getting things back how they should be currently. (Thanks for opening this @Daniel_Mietchen) TASK DETAIL https://phabricator.wikimedia.org/T290545 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Lydia_Pintscher, Ladsgroup, RKemper, Gnoeee, Fuzheado, Aklapper, Daniel_Mietchen, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280247: Create WCQS UI microsite deployment
RKemper added a comment. @Ladsgroup Puppet patch is merged now. I haven't merged the gui-deploy patches yet so those two gui-deploy patches should be good to go whenever now that the puppet changes are in place TASK DETAIL https://phabricator.wikimedia.org/T280247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, RKemper Cc: RKemper, Manuel, Ladsgroup, Addshore, Aklapper, Gehel, CBogen, ttaylor, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280006: Set up the application authentication for WCQS on commons-query.wikimedia.org
RKemper added a comment. Zbyszko, Erik, and I discussed the current status of this effort. We've realized that there's some open questions around how to actually implement the oauth, particularly with stuff like the callback URL given the way our routing logic works currently. Currently, for WDQS (which does not use oauth), users visit the "microsite" for the UI, but their requests end up hitting nginx (which forwards them to blazegraph). For WCQS, there won't be that pattern of a request hitting blazegraph directly. They'll be talking to the microsite, and the user is never going to do an interactive request against the backend server (e.g. nginx->blazegraph). ATS at the edge [*before* the request hits the internal loadbalancer (lvs)] will look at request and decide, based off the directory, whether it should go to the microsite or nginx. There's one special directory that will route to nginx, whereas everything else will go to the microsite. Where this breaks down is with oauth in the picture, this pattern won't work. - One approach would be for all traffic to route to nginx, and then have nginx decide what to forward to the microsite. As a general rule we were trying to avoid this approach though because we want this routing logic handled at the "infra" level, not the app layer. We'll need to figure out an approach here. TASK DETAIL https://phabricator.wikimedia.org/T280006 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, hnowlan, sdkim, BPirkle, Zache, Aklapper, Gehel, CBogen, ttaylor, Zbyszko, Invadibot, MPhamWMF, maantietaja, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment. Update: `sdcquery01` has been deleted, so we can proceed with removing the `Optional` type TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment. With respect to https://gerrit.wikimedia.org/r/c/operations/puppet/+/713946, we'll need to know if `sdcquery01` is still in use. If it's not and we can "decommission" it (or at least not worry about breaking it), that will let us require `oauth_settings` to be defined here, instead of being an `Optional`: https://github.com/wikimedia/puppet/blob/ae7ad4bc10f65ce260c623a1f4c06835d26f1c52/modules/profile/manifests/query_service/wcqs.pp#L21 (This is because for WCQS we always want `oauth_settings` to be set, so we want to disallow the use of null/undefined) TASK DETAIL https://phabricator.wikimedia.org/T280001 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: EBernhardson, Aklapper, Zbyszko, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280001: Set up puppet configuration for new WCQS cluster
RKemper added a comment. Log of steps to generate tls certs for wcqs with cergen for https://gerrit.wikimedia.org/r/713958 - `/srv/private` commit + 2 gerrit patches for TLS certs: --- (1) `[/srv/private] Commit SHA => a761d97c6395aab546d0024c4fce1b7174f50aa0` (2) `[operations/puppet] Patch => https://gerrit.wikimedia.org/r/c/operations/puppet/+/715569` (3) `[labs/private] Patch => https://gerrit.wikimedia.org/r/c/labs/private/+/715570` --- Log of steps to generate tls certs for wcqs with cergen for https://gerrit.wikimedia.org/r/713958 - **(1)** Create `/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/wcqs.certs.yaml` on puppetmaster1001.eqiad.wmnet Here's what it looks like for WDQS currently: ryankemper@puppetmaster1001:~$ cat /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml wdqs.discovery.wmnet: authority: puppet_ca expiry: null alt_names: ["wdqs.discovery.wmnet","wdqs.svc.eqiad.wmnet","wdqs.svc.codfw.wmnet","wdqs.wikimedia.org","wdqs1005.eqiad.wmnet","query.wikidata.org"] key: password: REDACTED algorithm: ec So let's try the following for wcqs: wcqs.discovery.wmnet: authority: puppet_ca expiry: null alt_names: ["wcqs.discovery.wmnet","wcqs.svc.eqiad.wmnet","wcqs.svc.codfw.wmnet","commons-query.wikimedia.org"] key: password: REDACTED algorithm: ec `sudo git add modules/secret/secrets/certificates/certificate.manifests.d/wcqs.certs.yaml` **(2)** `sudo cergen -c 'wcqs.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d` `=>` ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wcqs.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d 2021-08-30 17:26:10,965 INFO cergen Generating certificates ['wcqs.discovery.wmnet'] with force=False 2021-08-30 17:26:10,965 INFO Certificate(wcqs.discovery.wmnet) Generating all files, force=False... 2021-08-30 17:26:10,967 INFO Certificate(wcqs.discovery.wmnet) Generating certificate file /usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning /usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning /usr/lib/python3/dist-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning 2021-08-30 17:26:12,621 INFO Certificate(wcqs.discovery.wmnet) Generating CA certificate file 2021-08-30 17:26:12,622 INFO Certificate(wcqs.discovery.wmnet) Generating PKCS12 keystore file 2021-08-30 17:26:12,951 INFO Certificate(wcqs.discovery.wmnet) Generating Java keystore file 2021-08-30 17:26:13,931 INFO Certificate(wcqs.discovery.wmnet) Importing PuppetCA(puppetmaster1001.eqiad.wmnet_8140) cert into Java keystore 2021-08-30 17:26:15,092 INFO Certificate(wcqs.discovery.wmnet) Generating Java truststore file with CA certificate PuppetCA(puppetmaster1001.eqiad.wmnet_8140) Status of certificates ['wcqs.discovery.wmnet'] Certificate(wcqs.discovery.wmnet, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]): /srv/private/modules/secret/secrets/certificates/wcqs.discovery.wmnet/wcqs.discovery.wmnet.key.private.pem: PRESENT (mtime: 2021-08-30T17:26:10.966004) /srv/private/modules/secret/secrets/certificates/wcqs.discovery.wmnet/wcqs.discovery.wmnet.key.public
[Wikidata-bugs] [Maniphest] T207676: Parallelize scap deployment of WDQS
RKemper added a comment. I'll talk to rel-eng to see what scap changes are needed to parallelize between groups (wdqs eqiad public vs wdqs eqiad internal, etc) There's a chance it might be worth it to rely on a cookbook to rolling restart. Basically we'd use scap to get the new code in place and a cookbook to do the actual rolling restarts to actually uptake the changes. But for now I'd assume we'll just be changing it in scap-land and not introducing a cookbook TASK DETAIL https://phabricator.wikimedia.org/T207676 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Smalyshev, Gehel, Mathew.onipe, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
RKemper added a subtask: T289077: Migrate wdqs prometheus-based alerts from Icinga to alert-manager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Jdlrobson, fgiunchedi, Aklapper, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288501: Blazegraph journal too large on wdqs2003
RKemper claimed this task. RKemper edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T288501 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, RKemper, ema, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T288501: Blazegraph journal too large on wdqs2003
RKemper added a comment. ryankemper@wdqs2003:~$ ls -lah /srv/wdqs total 2.8T drwxrwxr-x 3 blazegraph wikidev4.0K Aug 6 07:26 . drwxr-xr-x 5 root root 4.0K Jun 2 22:59 .. -rw-rw-r-- 1 blazegraph wikidev 54 Aug 6 07:26 aliases.map -rw-rw-r-- 1 blazegraph blazegraph 23G Aug 10 07:41 categories.jnl -rw-r--r-- 1 root root 0 Jun 3 01:25 data_loaded -rw-r--r-- 1 blazegraph blazegraph 83K Aug 6 07:26 dcatap-20210806.rdf drwxrwxr-x 2 blazegraph wikidev 68K Aug 10 07:41 dumps -rw-rw-r-- 1 blazegraph blazegraph 2.8T Aug 9 19:25 wikidata.jnl ryankemper@wdqs2003:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs13G 1.3G 12G 11% /run /dev/md0 73G 15G 55G 22% / tmpfs63G 4.0K 63G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs63G 0 63G 0% /sys/fs/cgroup tmpfs 1.0G 20K 1.0G 1% /var/lib/nginx /dev/md22.9T 2.8T 120K 100% /srv tmpfs13G 0 13G 0% /run/user/24320 `wikidata.jnl` is over-inflated. here's a healthy host to compare to: ryankemper@wdqs2002:~$ ls -lah /srv/wdqs total 998G drwxrwxr-x 3 blazegraph wikidev4.0K Aug 6 07:38 . drwxr-xr-x 5 root root 4.0K Jun 4 04:46 .. -rw-rw-r-- 1 blazegraph wikidev 54 Aug 6 07:38 aliases.map -rw-rw-r-- 1 blazegraph blazegraph 23G Aug 10 07:56 categories.jnl -rw-r--r-- 1 root root 0 Jun 4 06:46 data_loaded -rw-r--r-- 1 blazegraph blazegraph 83K Aug 6 07:38 dcatap-20210806.rdf drwxrwxr-x 2 blazegraph wikidev 64K Aug 10 07:56 dumps -rw-rw-r-- 1 blazegraph blazegraph 975G Aug 10 17:07 wikidata.jnl ryankemper@wdqs2002:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 48G 0 48G 0% /dev tmpfs 9.5G 952M 8.6G 10% /run /dev/md0 73G 7.3G 62G 11% / tmpfs48G 4.0K 48G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs48G 0 48G 0% /sys/fs/cgroup tmpfs 1.0G 20K 1.0G 1% /var/lib/nginx /dev/md22.9T 999G 1.8T 36% /srv tmpfs 9.5G 0 9.5G 0% /run/user/24320 --- Solution is pretty simple, nuke `wikidata.jnl` (w/ the host de-pooled) and then kick off a `data-transfer` from `wdqs2002` (external) or `wdqs2005` (internal) TASK DETAIL https://phabricator.wikimedia.org/T288501 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, RKemper, ema, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T284445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T284445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T284445: Blazegraph journal too large on wdqs1012
RKemper moved this task from Ready for Development to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. This is done. New `wikidata.jnl` is 975G as expected: `-rw-rw-r-- 1 blazegraph blazegraph 975G Jun 8 19:01 wikidata.jnl` TASK DETAIL https://phabricator.wikimedia.org/T284445 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007
RKemper added a comment. Re-image of `wdqs2007` was completed successfully. TASK DETAIL https://phabricator.wikimedia.org/T281504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007
RKemper closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T281504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T281504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. We can't proceed any further on this ticket until we address https://phabricator.wikimedia.org/T280382, because the short-term fix for disk space will require re-imaging the affected `wdqs` hosts which will require starting the `data-transfer` process over (not the `data-reload`, fortunately). TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag
RKemper added a comment. 100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'systemctl start ...art wdqs-updater'. 100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. [1/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: 'NaN' [2/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '4458.16434332' [3/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '4230.269809265' [4/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '3850.3779346' [5/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '3175.483400543' [6/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '888.591801086' [7/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '39.694332428' [8/1000, retrying in 600.00s] Attempt to run 'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with base 10: '42.79952316284' Another small issue. Looks like we need to tell it to truncate (or just use doubles instead of ints) TASK DETAIL https://phabricator.wikimedia.org/T280108 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag
RKemper added a comment.
(For future reference)
import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
spicerack = Spicerack(verbose=True, dry_run=True)
def site(host):
"""Hacky way to get the site in which a host is located."""
if 'eqiad' in str(host):
return 'eqiad'
if 'codfw' in str(host):
return 'codfw'
raise ValueError('Site is unknown for {host}.'.format(host=host))
remote = spicerack.remote()
prometheus = spicerack.prometheus()
remote_host = remote.query('wdqs2008.codfw.wmnet')
host = remote_host.hosts[0].split(".")[0]
query = "scalar(time() - blazegraph_lastupdated{instance='%s:9193'})" % host
result = prometheus.query(query, site(remote_host))
last_updated = int(result[1])
if last_updated > 1200:
raise ValueError("Let's wait for updater to catch up, current
last_updated of {} is too high.".format(last_updated))
Snippet used to diagnose the above issue without having to actually run the
cookbook
TASK DETAIL
https://phabricator.wikimedia.org/T280108
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule,
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD,
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420,
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas,
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. Since we've seeded to `wdqs1004.eqiad.wmnet` we can start using that to transfer to other eqiad public nodes. Still need to seed to the other 3 (codfw public, codfw internal, eqiad internal) before we'll be at the peak "velocity". TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag
RKemper added a comment. Tested out the new cookbook, there's at least one small issue: Exception raised while executing cookbook sre.wdqs.data-transfer: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run raw_ret = runner.run() File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, in run return self._run(self.args, self.spicerack) File "/srv/deployment/spicerack/cookbooks/sre/wdqs/data-transfer.py", line 179, in run wait_for_updater(prometheus, site(source), source) File "/usr/lib/python3/dist-packages/spicerack/decorators.py", line 104, in wrapper return func(*args, **kwargs) # type: ignore File "/srv/deployment/spicerack/cookbooks/sre/wdqs/__init__.py", line 25, in wait_for_updater last_updated = int(result['value'][1]) TypeError: list indices must be integers or slices, not str END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) Error seems quite straightforward - we'll want to patch this. TASK DETAIL https://phabricator.wikimedia.org/T280108 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. I was having trouble understanding what the restart status of blazegraph has to do with the transfer of `/srv/wdqs/wikidata.jnl`. (As an aside, the point that running multiple simultaneous transfers from the same source is bottlenecked by outgoing bandwidth anyway makes perfect sense to me). It just clicked for me though - the cookbook transfers the file while blazegraph is disabled, so if another cookbook instance re-enables blazegraph while another cookbook instance is still receiving `/srv/wdqs/wikidata.jnl`, the `wikidata.jnl` will be getting mutated during the transfer. Note: I updated my comment above to indicate that `wdqs100[3,4]` are not properly done due to the data corruption. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. [EQIAD PUBLIC] DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1004.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1005.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1006.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1007.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1012.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1013.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> [CODFW PUBLIC] DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2001.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2002.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> [EQIAD INTERNAL] DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1003.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1008.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> [CODFW INTERNAL] DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2004.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2005.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "transferring wikidata journal following reload from dumps" --blazegraph_instance blazegraph --task-id T267927 <https://phabricator.wikimedia.org/T267927> TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nanda
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. `wdqs1009`, `wdqs1010`, and `wdqs2008` are done, so we need to `data-transfer` to the remaining instances. It looks like `wdqs1010` is warning (not critical) for disk space on `/srv`. Glancing at the box it's still got 44G left but is at 96% utilization: `/dev/mapper/vg0-srv 1.1T 999G 44G 96% /srv` This is essentially entirely attributable to `wikidata.jnl` which is **1 TB** in size. TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: ayounsi, jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment.
**DNS change logs**
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...
templates/wikidata.org | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
; Servers (alphabetic order)
; Service aliases
-query 1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.
; Wikis (alphabetic order)
Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
templates/wikidata.org | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
-- Generating zonefiles from zone templates
-- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
-- Copying automatically generated zone files under target tree
-- Copying repo-driven real config files and admin_state
-- Copying puppetized config and GeoIP from /etc/gdnsd
-- Checking for illegal tabs in zonefiles
-- Running zone_validator to check WMF rules
-- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
-- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
-- Descending to subdirectory: netbox
-- Done with subdir: netbox
-- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---
authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
(11)
---
OK - authdns updated successfully
OK - authdns-update successful on all nodes!
Validation looks good:
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done
; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE rcvd: 84
; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE rcvd: 84
; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE rcvd: 84
TASK DETAIL
https://phabricator.wikimedia.org/T266470
EMAIL PREFERENCES
https://phabricator.wikimedi
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. The issues with envoy were resolved by running `sudo /usr/local/sbin/build-envoy-config -c /etc/envoy` to properly build `/etc/envoy/envoy.yaml`. That should have been done by puppet already, triggered upon a `sudo systemctl restart envoyproxy.service`, but it didn't - perhaps a race condition. See https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/init.pp#81 and https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/conf.pp#30 for the puppet code that normally does it automatically. **Envoy is working properly now and wdqs1009 is now reachable from the caching layer** TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment.
**DNS change logs**
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...
templates/wikidata.org | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
; Servers (alphabetic order)
; Service aliases
-query 1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.
; Wikis (alphabetic order)
Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
templates/wikidata.org | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
-- Generating zonefiles from zone templates
-- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
-- Copying automatically generated zone files under target tree
-- Copying repo-driven real config files and admin_state
-- Copying puppetized config and GeoIP from /etc/gdnsd
-- Checking for illegal tabs in zonefiles
-- Running zone_validator to check WMF rules
-- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
-- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
-- Descending to subdirectory: netbox
-- Done with subdir: netbox
-- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---
authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
(11)
---
OK - authdns updated successfully
OK - authdns-update successful on all nodes!
Validation looks good:
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done
; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE rcvd: 84
; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE rcvd: 84
; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN ANY
;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME dyna.wikimedia.org.
;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE rcvd: 84
TASK DETAIL
https://phabricator.wikimedia.org/T266470
EMAIL PREFERENCES
https://phabricator.wikimedi
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. I missed a step yesterday: I'd updated `/srv/private` as well as the public `labs/private` repo but missed the step for updating `operations/puppet` with the new pubkey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671267 TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Current status for when I pick this back up: - `query-preview.wikidata.org` (with `wdqs1009.eqiad.wmnet` as an `alt_name`) is generated - reverted https://gerrit.wikimedia.org/r/c/operations/puppet/+/670339/ for now, envoy wasn't happy: Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: Failed to call refresh: '/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 instead of one of [0] Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: '/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 instead of one of [0] Error: /Stage[main]/Profile::Tlsproxy::Envoy/Sslcert::Certificate[query-preview.wikidata.org]/File[/etc/ssl/localcerts/query-preview.wikidata.org.crt]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///files/ssl/query-preview.wikidata.org.crt TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Ah, so poking around the `certificate.manifests.d` repo I see certs that don't necessarily follow the `discovery.wmnet` pattern. To me that implies **Option 2** should be working, so I might be missing something. Here's an example that doesn't use discovery: ryankemper@puppetmaster1001:/srv/private$ cat modules/secret/secrets/certificates/certificate.manifests.d/analytics_http_ui.certs.yaml yarn.wikimedia.org: authority: puppet_ca expiry: null alt_names: ["yarn.wikimedia.org", "hue.wikimedia.org", "hue-next.wikimedia.org", "superset.wikimedia.org", "pivot.wikimedia.org", "turnilo.wikimedia.org", "stats.wikimedia.org", "analytics.wikimedia.org", "piwik.wikimedia.org", "datasets.wikimedia.org"] key: password: REDACTED algorithm: ec TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. **Option 2 ** fails to even generate the cert. All the cergen documentation is written for a certificate like `query-preview.discovery.wmnet` and not `wdqs1009.eqiad.wmnet` or `query-preview.wikidata.org`. So I do think this just isn't what cergen is built to do. ryankemper@puppetmaster1001:/srv/private$ git status On branch master Untracked files: (use "git add ..." to include in what will be committed) modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml nothing added to commit but untracked files present (use "git add" to track) ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d 2021-03-12 08:06:03,210 INFO cergen Generating certificates [] with force=False Status of certificates [] TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment.
Hit a big blocker with the current proposed approach of using
`wdqs1009.eqiad.wmnet` as the cert name:
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*'
--generate --base-path /srv/private/modules/secret/secrets/certificates
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-03-12 07:55:03,774 INFO cergen
Generating certificates ['wdqs1009.eqiad.wmnet'] with force=False
2021-03-12 07:55:03,774 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating all files, force=False...
2021-03-12 07:55:03,776 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating certificate file
/usr/lib/python3/dist-packages/urllib3/connection.py:362:
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no
`subjectAltName`, falling back to check for a `commonName` for now. This
feature is being removed by major browsers and deprecated by RFC 2818. (See
https://github.com/shazow/urllib3/issues/497 for details.)
SubjectAltNameWarning
Traceback (most recent call last):
File "/usr/bin/cergen", line 11, in
load_entry_point('cergen==0.2.5', 'console_scripts', 'cergen')()
File "/usr/lib/python3/dist-packages/cergen/main.py", line 93, in main
certificate.generate(force=args['--force'])
File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 304, in
generate
self.generate_crt(force=force)
File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 344, in
generate_crt
self.x509_cert = self.authority.sign(csr, self.expiry)
File "/usr/lib/python3/dist-packages/cergen/puppet.py", line 193, in sign
self.name, common_name
RuntimeError: puppetmaster1001.eqiad.wmnet_8140 already has a signed
certificate for wdqs1009.eqiad.wmnet. If you are trying to regenerate a Puppet
CA signed certificate, you need to first remove the certificate from the Puppet
CA. On the puppetmaster, `puppet cert clean wdqs1009.eqiad.wmnet` should do it.
Corresponding to this cergen manifest file
`modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml`:
wdqs1009.eqiad.wmnet:
authority: puppet_ca
expiry: null
alt_names: ["wdqs1009.eqiad.wmnet","query-preview.wikidata.org"]
key:
password: REDACTED
algorithm: ec
I'm a bit lost on the right way to proceed. Thinking out loud here:
**Option 1** Just actually re-generate the cert with `puppet cert clean
wdqs1009.eqiad.wmnet` before running cergen. This feels wrong though, and I
think ultimately it comes down to using cergen in a way that wasn't intended:
name collision with the existing cert puppet has.
**Option 2** Do this instead:
query-preview.wikidata.org:
authority: puppet_ca
expiry: null
alt_names: ["query-preview.wikidata.org", "wdqs1009.eqiad.wmnet"]
key:
password: REDACTED
algorithm: ec
Might want to talk to traffic again. But in the interest of iteration speed I
think trying out Option **2** is not a bad idea; there's not really any risk
since this is pertaining to `wdqs1009.eqiad.wmnet` which is a test host.
TASK DETAIL
https://phabricator.wikimedia.org/T266470
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja,
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301,
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30,
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic,
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen,
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas,
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Finished rolling back to the previous iteration of `wdqs.discovery.wmnet` cert since we're now going to create a net-new cert `wdqs1009.eqiad.wmnet` for `wdqs-test` TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Certificate(wdqs.discovery.wmnet, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]): /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem: PRESENT (mtime: 2019-10-21T08:11:52.741452) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem: PRESENT (mtime: 2019-10-21T08:11:52.745452) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem: PRESENT (mtime: 2021-03-10T19:17:36.206506) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem: PRESENT (mtime: 2019-12-10T14:26:42.496697) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12: PRESENT (mtime: 2021-03-10T19:17:36.226506) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks: PRESENT (mtime: 2021-03-10T19:17:38.030503) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks: PRESENT (mtime: 2021-03-10T19:17:38.938502) ryankemper@puppetmaster1001:/srv/private$ git status On branch master Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12 TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Certificate(wdqs.discovery.wmnet, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]): /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem: PRESENT (mtime: 2019-10-21T08:11:52.741452) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem: PRESENT (mtime: 2019-10-21T08:11:52.745452) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem: PRESENT (mtime: 2021-03-10T04:58:24.029629) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem: PRESENT (mtime: 2019-12-10T14:26:42.496697) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12: PRESENT (mtime: 2021-03-10T04:58:24.049629) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks: PRESENT (mtime: 2021-03-10T04:58:25.893621) /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks: PRESENT (mtime: 2021-03-10T04:58:26.865617) ryankemper@puppetmaster1001:/srv/private$ git status On branch master Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks modified: modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12 no changes added to commit (use "git add" and/or "git commit -a") Relevant status after performing the cergen. (Git commit not created yet as seen by the dirty working directory) TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T252504: Automate the smoke test of the canary deployment of WDQS
RKemper renamed this task from "Smoke test the canary deployment of WDQS" to "Automate the smoke test of the canary deployment of WDQS". RKemper updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T252504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, Zbyszko, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Here's how the ats mapping looks afterdeploy of the `backend.yaml` changes: ryankemper@cp1075:~$ sudo cat /etc/trafficserver/remap.config | grep query map http://query-preview.wikidata.org/bigdata https://wdqs1009.eqiad.wmnet/bigdata map http://query-preview.wikidata.org/sparql https://wdqs1009.eqiad.wmnet/sparql map http://query-preview.wikidata.org https://webserver-misc-apps.discovery.wmnet map http://query.wikidata.org/bigdata/ldf https://wdqs1005.eqiad.wmnet/bigdata/ldf map http://query.wikidata.org/sparql https://wdqs.discovery.wmnet/sparql map http://query.wikidata.org/bigdata https://wdqs.discovery.wmnet/bigdata map http://query.wikidata.org https://webserver-misc-apps.discovery.wmnet TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. Posting logs of our IRC convo from ~1 month ago for context when I tag people for review: [2021-02-04 11:31:36] bblack: around ? [2021-02-04 11:31:58] I'm here with ryankemper to talk about T266470, unless you want to jump in a meet [2021-02-04 11:31:59] T266470: Expose wdqs1009 to wdqs users and gather feedback - https://phabricator.wikimedia.org/T266470 [2021-02-04 11:32:51] taking notes in https://etherpad.wikimedia.org/p/expose-wdqs1009 [2021-02-04 11:33:08] gehel: hi :) [2021-02-04 11:33:15] o/ [2021-02-04 11:33:15] ryankemper: hi too :) [2021-02-04 11:33:23] \o [2021-02-04 11:33:40] so, I guess my best understanding of the present stuff is what I see in: [2021-02-04 11:33:47] context: we want to expose wdqs1009 as a test server so that our users can make sure we're not breaking anything with the new WDQS updater [2021-02-04 11:33:53] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#157 [2021-02-04 11:34:12] there's several related definitions there for https://query.wikidata.org/ [2021-02-04 11:34:24] for different subspaces of the URI routing to different backends [2021-02-04 11:34:40] /bigdata/ldf goes explicitly to wds1005 [2021-02-04 11:35:07] /bigdata and /sparql both go to the discovery service to a set of wdqs [2021-02-04 11:35:25] but the root url hits our misc webapps setup, for the JS powering the query UI [2021-02-04 11:35:29] yep, somewhat similar, but in this case, we want to have a different FQDN (probably query-preview.wikidata.org or something similar) and route to a different server that isn't part of the current WDQS pools [2021-02-04 11:36:45] so in this case, I would guess, you'd want no special exception for /bigdata/ldf, and you'd want all of /bigdata and /sparql -> wdqs1009 [2021-02-04 11:37:06] which is something we can just define in that file I linked, and adding a name at the DNS layer, and done [2021-02-04 11:37:16] but the root URL for the UI part, might be a little trickier [2021-02-04 11:37:39] I need to see where that's defined in git, and it might need a new site definition, and to template in a new place to send the UI queries... [2021-02-04 11:37:57] but there's no other special requirements I think/expect? [2021-02-04 11:37:57] we'll probably want to create another microsite for the JS part, that's fairly straightforward [2021-02-04 11:38:15] no other special requirements IIRC [2021-02-04 11:38:17] yep, that seems fairly simple to me [2021-02-04 11:38:22] so to resume: [2021-02-04 11:39:08] create a DNS entry (CNAME to dyna.wm.o), another set of entries in backend.yaml map, create another minisite (with the appropriate configuration) [2021-02-04 11:39:22] do we need to do anything for SSL certs? Or is that fully automated? [2021-02-04 11:39:48] that part's all shared between everything that runs through the cache, so long as you stick to our canonical domains and just change the leading hostname part [2021-02-04 11:40:04] to e.g. query-preview.wikidata.org [2021-02-04 11:40:05] "canonical" includes wikidata.org? [2021-02-04 11:40:11] ok, good! [2021-02-04 11:40:37] yeah for reference, the canonical set is: https://wikitech.wikimedia.org/wiki/HTTPS#For_the_Foundation's_canonical_domain_names [2021-02-04 11:41:13] ok, one last stupid and off topic question: what does "dyna" stands for in dyna.wm.o? [2021-02-04 11:42:14] it's not a "real" DNS RR-type you would see in the wild. It's a custom thing our authserver implements which provides dynamic A and records depending on geodns. that's how we route to the closest of our geographic edges for users [2021-02-04 11:42:36] https://github.com/gdnsd/gdnsd/wiki/GdnsdPluginGeoip [2021-02-04 11:42:41] ^ that, basically [2021-02-04 11:42:57] thanks! [2021-02-04 11:43:23] templates/wikimedia.org:dyna600 IN DYNA geoip!text-addrs [2021-02-04 11:43:36] ^ is what all the other production CNAMEs are pointing towards [2021-02-04 11:43:54] gehel: also I don't know if you've created new service hostnames since the move to netbox [2021-02-04 11:44:03] but we don't do commits on the ops/dns repo for that kind of thing anymore [2021-02-04 11:44:25] Oh no, I havent [2021-02-04 11:44:33] ryankemper: is the one who's goign to do it [2021-02-04 11:45:00] I assume it's documented [2021-02-04 11:45:01] oh wait, I'm getting ahead of ourselves [2021-02-04 11:45:03] — gehel is searching [2021-02-04 11:45:13] that part still is done in ops/dns ! :) [2021-02-04 11:45:25] now I'm curious anyway! [2021-02-04 11:45:43] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. Note: Puppet is still disabled on `wdqs2008` while the reload runs. It occurred to me that I'm not sure if puppet actually **needs** to be disabled during data reloads or if that's just a precaution we've historically taken - any insight here @Gehel? TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T274751: Upgrade firmware on wdqs1009
RKemper added a comment. @Cmjohnson The data reload is complete on `wdqs1009`, so the host can now have its firmware upgraded and be rebooted at its convenience. Note this is an internal wdqs test host, so there is no public-facing service for us to worry about. Feel free to proceed when convenient for you - just shoot me a ping or drop a line in #wikimedia-discovery, but no need to schedule a formal window. TASK DETAIL https://phabricator.wikimedia.org/T274751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, RKemper Cc: Jclark-ctr, RKemper, Cmjohnson, Gehel, MoritzMuehlenhoff, Ramtin0071, MPhamWMF, Devnull, lmata, Muchiri124, CBogen, Akuckartz, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Addshore, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. Downtimed `wdqs2008` until `2021-03-04 21:56:59` TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment. Still waiting for the latest dumps to be downloaded (few more hours), then need to reboot WDQS hosts as part of https://phabricator.wikimedia.org/T274213, then can do the actual data-reload TASK DETAIL https://phabricator.wikimedia.org/T267927 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink
RKemper added a comment. (See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket has been closed) TASK DETAIL https://phabricator.wikimedia.org/T266495 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: RKemper, dcausse, elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big
RKemper added a comment. Updated: [X] https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates [X] https://commons.wikimedia.org/wiki/Commons:Village_pump#Unscheduled_maintenance%3A_Wikimedia_Commons_Query_Service [X] WikiData ML mailing list (note: A community member pointed out that sending an e-mail to a Commons mailing list would have made more sense, which is a great point. We'll do that in the future) TASK DETAIL https://phabricator.wikimedia.org/T273636 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. RKemper added a comment. WCQS is back in service; updating the notification channels right now and will comment back here after TASK DETAIL https://phabricator.wikimedia.org/T273636 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big
RKemper claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T273636 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps
RKemper added a comment.
`sudo cookbook sre.wdqs.data-reload wdqs1009.eqiad.wmnet
--reuse-downloaded-dump --reload-data wikidata --skolemize --reason 'T267927:
Reload wikidata jnl from fresh dumps' --task-id T267927 ` is failing with:
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -
- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -
- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -
- OUTPUT of 'rm -rf /srv/wdqs...d -- --skolemize' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -
%msg%n
22:46:13.025 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{'
[line 2]
at
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFParserHelper.java:440)
at
org.openrdf.rio.helpers.RDFParserBase.reportFatalError(RDFParserBase.java:685)
at
org.openrdf.rio.turtle.TurtleParser.reportFatalError(TurtleParser.java:1405)
at org.openrdf.rio.turtle.TurtleParser.parseValue(TurtleParser.java:662)
at
org.openrdf.rio.turtle.TurtleParser.parsePredicate(TurtleParser.java:505)
at
org.openrdf.rio.turtle.TurtleParser.parsePredicateObjectList(TurtleParser.java:420)
at
org.openrdf.rio.turtle.TurtleParser.parseImplicitBlank(TurtleParser.java:613)
at
org.openrdf.rio.turtle.TurtleParser.parseTriples(TurtleParser.java:394)
at
org.openrdf.rio.turtle.TurtleParser.parseStatement(TurtleParser.java:259)
at org.openrdf.rio.turtle.TurtleParser.parse(TurtleParser.java:214)
at org.wikidata.query.rdf.tool.Munge.run(Munge.java:105)
at org.wikidata.query.rdf.tool.Munge.main(Munge.java:59)
22:46:13.024
[org.wikidata.query.rdf.tool.rdf.AsyncRDFHandler$RDFActionsReplayer] INFO
org.wikidata.query.rdf.tool.Munge - Switching to
/srv/wdqs/munged/wikidump-1.ttl.gz
`sudo cookbook sre.wdqs.data-reload wdqs1010.eqiad.wmnet
--reuse-downloaded-dump --reload-data wikidata --reason 'T267927: Reload
wikidata jnl from fresh dumps' --task-id T267927`is failing with:
ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-reload
wdqs1010.eqiad.wmnet --reuse-downloaded-dump --reload-data wikidata --reason
'T267927: Reload wikidata jnl from fresh dumps' --task-id T267927
START - Cookbook sre.wdqs.data-reload
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -
PASS
||
100% (1/1) [00:00<00:00, 1.92hosts/s]
FAIL |
| 0% (0/1)
[00:00= 100.0% threshold) for command: 'test -f
/srv/wdq...test-all.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully
executed all commands.
Detected dump (/srv/wdqs/latest-all.ttl.bz2). Skipping download
- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -
PASS
||
100% (1/1) [00:00<00:00, 2.44hosts/s]
FAIL |
| 0% (0/1)
[00:00= 100.0% threshold) for command: 'test -f
/srv/wdq...-lexemes.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully
executed all commands.
Detected dump (/srv/wdqs/latest-lexemes.ttl.bz2). Skipping download
checking available disk space
- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -
PASS
||
100% (1/1) [00:00<00:00, 2.45hosts/s]
FAIL |
| 0% (0/1)
[00:00= 100.0% threshold) for command:
'dump_size=`du /s...avail+$db_size))'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully
executed all commands.
Running munger for main database and then lexeme
munging /srv/wdqs/munged (skolemizaton: False)
- OUTPUT of 'rm -rf /srv/wdqs.../wdqs/munged -- ' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -
%msg%n
22:50:03.137 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{'
[line 2]
at
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFPar
[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback
RKemper added a comment. TODO from IRC meeting with bblack/gehel: create a DNS entry (CNAME to dyna.wm.o), another set of entries in backend.yaml map, create another minisite (with the appropriate configuration) TASK DETAIL https://phabricator.wikimedia.org/T266470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
