[Wikidata-bugs] [Maniphest] T348831: [WD-ORG] [TECH] Max Lag alerts misfire with a DataSource error
fgiunchedi added a comment. FWIW I agree with testing the different boundaries, especially as you pointed out the alert is lax in terms of "reactivity" TASK DETAIL https://phabricator.wikimedia.org/T348831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Michael, fgiunchedi, Lucas_Werkmeister_WMDE, Aklapper, ItamarWMDE, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, lmata, Akuckartz, Nandana, colewhite, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T348831: [WD-ORG] [TECH] Max Lag alerts misfire with a DataSource error
fgiunchedi added a comment. For most cases I think alerting on "no data" and "values are all null" is sensible, in other words you expect to have data returned by the query at all times. In this case I can't quite figure out why the alert went "no data"; I've searched grafana logs for the rule uid though Nov 23 16:10:07 grafana1002 grafana[18790]: logger=ngalert.sender.router rule_uid=MF0FSjJ4z org_id=1 t=2023-11-23T16:10:06.999230004Z level=info msg="Sending alerts to local notifier" count=1 Nov 23 16:12:07 grafana1002 grafana[18790]: logger=ngalert.sender.router rule_uid=MF0FSjJ4z org_id=1 t=2023-11-23T16:12:07.2677782Z level=info msg="Sending alerts to local notifier" count=1 Nov 23 16:14:04 grafana1002 grafana[18790]: logger=ngalert.state.manager rule_uid=MF0FSjJ4z org_id=1 t=2023-11-23T16:14:04.878943734Z level=info msg="Detected stale state entry" cacheID="[[\"__alert_rule_namespace_uid__\",\"k0zbgDsik\"],[\"__alert_rule_uid__\",\"MF0FSjJ4z\"],[\"__contacts__\",\"\\\"AlertManager\\\"\"],[\"alertname\",\"DispatchChanges Normal job backlog time (p50, 15min) alert\"],[\"datasource_uid\",\"00026\"],[\"grafana_folder\",\"Wikidata\"],[\"ref_id\",\"A\"],[\"rule_uid\",\"MF0FSjJ4z\"],[\"severity\",\"critical\"],[\"team\",\"wikidata\"]]" state=Alerting reason=NoData Nov 23 16:14:04 grafana1002 grafana[18790]: logger=ngalert.sender.router rule_uid=MF0FSjJ4z org_id=1 t=2023-11-23T16:14:04.904922823Z level=info msg="Sending alerts to local notifier" count=1 TASK DETAIL https://phabricator.wikimedia.org/T348831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Lucas_Werkmeister_WMDE, Aklapper, ItamarWMDE, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, lmata, Akuckartz, Nandana, colewhite, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T350255: Repeated Wikidata Grafana alerts due to "failed to query data"
fgiunchedi added a comment. Thank you for reaching out @Lucas_Werkmeister_WMDE ! Yes indeed known issue, we (o11y) recommend turning off notifications for datasource errors (full rationale in https://phabricator.wikimedia.org/T347221#9264101) and the instructions being at https://wikitech.wikimedia.org/wiki/Grafana#DatasourceError_notification_spam TASK DETAIL https://phabricator.wikimedia.org/T350255 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, lmata, ItamarWMDE, Akuckartz, Nandana, colewhite, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281267: various weekly and daily dumps run from systemd timers are broken
fgiunchedi added a comment. In T281267#8954763 <https://phabricator.wikimedia.org/T281267#8954763>, @ArielGlenn wrote: > @fgiunchedi I notice that in some cases phab tasks are autocreated when systemd units fail. Is that true for systemd jobs on snapshot hosts? Could we get tagged on those (Dumps-Generation) or could we get emails from those (ops-dumps@wm.o)? Yes you can! The easiest would be to add a section to Alertmanager routing for `team=core-platform` alerts, and decide what to do depending on the alert and/or its severity. A good starting point is this: https://wikitech.wikimedia.org/wiki/Alertmanager#I'm_part_of_a_new_team_that_needs_onboarding_to_Alertmanager,_what_do_I_need_to_do? and please reach out if you run into any snags! TASK DETAIL https://phabricator.wikimedia.org/T281267 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Tonina_Zhelyazkova_WMDE, WMDE-leszek, JAllemandou, fgiunchedi, jbond, hoo, dcausse, ArielGlenn, Protsack.stephan, Busfault, Astuthiodit_1, Atieno, karapayneWMDE, joanna_borun, Invadibot, Devnull, maantietaja, lmata, Muchiri124, jannee_e, ItamarWMDE, Akuckartz, holger.knust, Legado_Shulgin, ReaperDawn, Nandana, Davinaclare77, Techguru.pc, Lahi, Gq86, herron, GoranSMilovanovic, Chicocvenancio, Lunewa, Hfbn0, QZanden, LawExplorer, Zppix, Volans, _jensen, rosalieper, Scott_WUaS, Wong128hk, gnosygnu, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Hokwelum ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T332953: Migrate PipelineLib repos to GitLab
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T332953 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, WMDE-leszek, leila, fkaelin, ItamarWMDE, elukey, KartikMistry, santhosh, Martaannaj, sbassett, bking, bd808, Ladsgroup, Krinkle, Legoktm, tstarling, Physikerwelt, dcausse, Jdrewniak, taavi, hnowlan, Michaelcochez, cjming, Jdforrester-WMF, dduvall, Aklapper, thcipriani, Bellucii32, Itsmeduncan, Cleo_Lemoisson, Astuthiodit_1, EChetty, TheReadOnly, karapayneWMDE, toberto, joanna_borun, Simonmaignan, Invadibot, MPhamWMF, Devnull, maantietaja, calbon, Muchiri124, Confetti68, Anerka, CBogen, Nintendofan885, Akuckartz, Otr500, WDoranWMF, Ddurigon, MJL, brennen, Mateo1977, EvanProdromou, Legado_Shulgin, ReaperDawn, Nandana, NebulousIris, Namenlos314, aezell, skpuneethumar, Zylc, Davinaclare77, Abdeaitali, 1978Gage2001, Techguru.pc, Lahi, Operator873, Gq86, Xinbenlv, Vacio, Sharvaniharan, Bsandipan, scblr, Xover, GoranSMilovanovic, SPoore, TBolliger, Chicocvenancio, Hfbn0, QZanden, EBjune, Tbscho, Taquo, LawExplorer, catalandres, Eginhard, Zppix, JJMC89, TerraCodes, DDJJ, _jensen, rosalieper, Agabi10, PEarleyWMF, RuyP, Liudvikas, Scott_WUaS, Pchelolo, Karthik_sripal, Izno, Wong128hk, Luke081515, Bsadowski1, Niharika, Wikidata-bugs, Jitrixis, aude, Bawolff, Dbrant, Dinoguy1000, Gryllida, Lydia_Pintscher, faidon, Grunny, ssastry, scfc, Alchimista, Arlolra, csteipp, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294014: Invalid wikidata daily metrics received
fgiunchedi removed a project: Observability-Metrics. fgiunchedi added a comment. Not a problem AFAICS, removing o11y TASK DETAIL https://phabricator.wikimedia.org/T294014 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Addshore, Lydia_Pintscher, Manuel, Lucas_Werkmeister_WMDE, Aklapper, fgiunchedi, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331, lmata, colewhite, herron ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316031: Clean up the rdf-streaming-updater-codfw container from thanos-swift.
fgiunchedi added a comment. Thank you for following up, I think the culprit is the fact that the S3 <https://phabricator.wikimedia.org/S3> compat API stores chunks of big files in a separate container (suffixed with `+segments`). See also the audit below I ran logged into swift as `wdqs:flink`: # swift list | xargs -n1 swift stat | grep -e Container -e Objects -e Bytes Container: rdf-streaming-updater-codfw Objects: 125 Bytes: 336259981 Container: rdf-streaming-updater-codfw+segments Objects: 3752575 Bytes: 19043034820255 Container: rdf-streaming-updater-codfw-T314835 Objects: 0 Bytes: 0 Container: rdf-streaming-updater-eqiad Objects: 3079 Bytes: 61140716537 Container: rdf-streaming-updater-eqiad+segments Objects: 13832 Bytes: 55454475470 Container: rdf-streaming-updater-staging Objects: 1423 Bytes: 5878764535 Container: thanos-swift Objects: 48 Bytes: 25856176 Container: updater Objects: 2921 Bytes: 86172646251 Container: updater+segments Objects: 552 Bytes: 2856364302 Container: updater-zbyszko Objects: 31 Bytes: 1622577 Container: updater-zbyszko-v2 Objects: 36 Bytes: 120047231 TASK DETAIL https://phabricator.wikimedia.org/T316031 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, fgiunchedi Cc: Aklapper, fgiunchedi, dr0ptp4kt, LSobanski, dcausse, Ottomata, elukey, gmodena, MatthewVernon, EBernhardson, bking, Raineydaz, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, Muchiri124, CBogen, ItamarWMDE, Akuckartz, Di3sel1975, Chambersjay, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Conradrock, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
fgiunchedi added a comment. In T314835#8178848 <https://phabricator.wikimedia.org/T314835#8178848>, @dcausse wrote: > Moving forward we will: > > - stop the presto-swift client in favor of an S3 <https://phabricator.wikimedia.org/S3> connector. > - cleanup the `rdf-streaming-updater-codfw` container > - monitor and alert on the space usage on these containers (if there's also way to implement a quota per container I'd be in favor of doing so) > > Unless I missed something or that we want to continue tracking some work with task I believe we can close this task. Thank you @dcausse for the write up and action items, all looks good to me! TASK DETAIL https://phabricator.wikimedia.org/T314835 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: EBernhardson, MatthewVernon, gmodena, elukey, bking, Ottomata, dcausse, LSobanski, dr0ptp4kt, fgiunchedi, Aklapper, Hellket777, Raineydaz, LisafBia6531, Astuthiodit_1, AWesterinen, 786, BTullis, Biggs657, karapayneWMDE, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Di3sel1975, Hook696, Kent7301, Chambersjay, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Conradrock, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
fgiunchedi closed subtask T314914: Bump memcache connections and swift-proxy limits as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T314835 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: MatthewVernon, gmodena, elukey, bking, Ottomata, dcausse, LSobanski, dr0ptp4kt, fgiunchedi, Aklapper, Hellket777, Raineydaz, LisafBia6531, Astuthiodit_1, AWesterinen, 786, BTullis, Biggs657, karapayneWMDE, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Di3sel1975, Hook696, Kent7301, Chambersjay, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Conradrock, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
fgiunchedi added a comment. In T314835#8141914 <https://phabricator.wikimedia.org/T314835#8141914>, @fgiunchedi wrote: > Thank you @dcausse for diving deep into this issue and mitigating it! I can confirm that the space has stopped growing at the same rate (i.e. not growing ATM). > > I can confirm that I've seen the same failures from swift client doing mass deletes, not sure why though. I noticed this independently when trying to delete big Tegola containers in T307184: Followups for Tegola and Swift interactions <https://phabricator.wikimedia.org/T307184>, while some deletes timeout the swift build delete continues with the remaining files. I think once the first pass of deletes is done then it'd be sufficient to repeat the command as many times as needed > I am looking into the auth failures in codfw and can confirm that too, only on thanos-fe2001 though! I have depooled that host as a precaution for now I have mitigated the auth failures for now (permanent fix in https://phabricator.wikimedia.org/T314914) TASK DETAIL https://phabricator.wikimedia.org/T314835 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: gmodena, elukey, bking, Ottomata, dcausse, LSobanski, dr0ptp4kt, fgiunchedi, Aklapper, Hellket777, Raineydaz, LisafBia6531, Astuthiodit_1, AWesterinen, 786, MatthewVernon, BTullis, Biggs657, karapayneWMDE, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Di3sel1975, Hook696, Kent7301, Chambersjay, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Conradrock, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
fgiunchedi added a comment. Thank you @dcausse for diving deep into this issue and mitigating it! I can confirm that the space has stopped growing at the same rate (i.e. not growing ATM). I can confirm that I've seen the same failures from swift client doing mass deletes, not sure why though. I am looking into the auth failures in codfw and can confirm that too, only on thanos-fe2001 though! I have depooled that host as a precaution for now TASK DETAIL https://phabricator.wikimedia.org/T314835 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: gmodena, elukey, bking, Ottomata, dcausse, LSobanski, dr0ptp4kt, fgiunchedi, Aklapper, Hellket777, Raineydaz, LisafBia6531, Astuthiodit_1, AWesterinen, 786, MatthewVernon, BTullis, Biggs657, karapayneWMDE, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Di3sel1975, Hook696, Kent7301, Chambersjay, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Conradrock, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
fgiunchedi created this task. fgiunchedi added projects: Wikidata-Query-Service, SRE, SRE-swift-storage. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION It looks like wdqs more than tripled its storage space usage in the span of 10 days (from ~6T to ~21T), is this expected? We should cull its disk usage or risk running out of disk space on the whole thanos-swift cluster see also the account's space usage: https://thanos.wikimedia.org/graph?g0.expr=swift_account_stats_bytes_total%7Baccount%3D%22AUTH_wdqs%22%7D=0=0_input=2w_source_resolution=0s=1_response=0_matches=%5B%5D TASK DETAIL https://phabricator.wikimedia.org/T314835 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Raineydaz, AWesterinen, MatthewVernon, joanna_borun, Lalamarie69, MPhamWMF, Devnull, LSobanski, Muchiri124, CBogen, Di3sel1975, Chambersjay, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, Conradrock, Davinaclare77, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Hfbn0, EBjune, merbst, Zppix, elukey, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi removed a subtask: T300723: Migrate Traffic Prometheus alerts from Icinga to Alertmanager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi removed a subtask: T294564: Migrate Foundations Prometheus alerts to AlertManager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi removed a subtask: T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi removed a subtask: T289077: Migrate Search team's prometheus-based alerts from Icinga to alert-manager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi removed a subtask: T285328: Migrate OSM sync alerts from icinga to AlertManager. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi closed this task as "Resolved". fgiunchedi claimed this task. fgiunchedi added a comment. Resolving this in favor of parent task TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi edited projects, added SRE Observability (FY2022/2023-Q1); removed SRE Observability (FY2021/2022-Q4). TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, colewhite, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T297494: Port Wikidata dashboard data from Graphite to Prometheus
fgiunchedi added a comment. In T297494#8036682 <https://phabricator.wikimedia.org/T297494#8036682>, @ItamarWMDE wrote: >> My understanding is that you are solving for the former problem (i.e. MW) (?) > > In this particular case, yes, the metrics are collected from Wikidata itself (please correct me if I'm wrong @Manuel). Thank you for confirming! > However, more generally speaking, we do have some tools surrounding wikidata, that are not MW based (predominantly hosted in various toolforge containers), in which we still used statsv/statsd. Is there any clear guide on how to migrate away from these methods in those projects? I can't speak for toolforge (though I believe there are Prometheus solutions there too!). However for production yes, the basic methods are outlined here: https://wikitech.wikimedia.org/wiki/Prometheus#Adding_new_metrics . For statsd specifically the basic idea is illustrated here: https://wikitech.wikimedia.org/wiki/Prometheus#Statsd and the k8s-specific documentation can be found at https://wikitech.wikimedia.org/wiki/Prometheus/statsd_k8s I'm sure there are gaps in the documentation, please reach out with any other questions/doubts/etc ! TASK DETAIL https://phabricator.wikimedia.org/T297494 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ItamarWMDE, fgiunchedi Cc: Krinkle, colewhite, fgiunchedi, Aklapper, Manuel, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T297494: Port Wikidata dashboard data from Graphite to Prometheus
fgiunchedi added subscribers: colewhite, Krinkle. fgiunchedi added a comment. In T297494#8022604 <https://phabricator.wikimedia.org/T297494#8022604>, @ItamarWMDE wrote: > Reposting from a Slack discussion, it appears as though statsd is still the preferred way to gather some metrics (as opposed to ephemeral jobs), is that correct @fgiunchedi? In that case, is it our responsibility to set up export of these into Prometheus with statsd exporter? Which data collection methods could be migrated to Prometheus otherwise? Is there a place where we can find a more detailed migration guideline from Graphite to Prometheus to instruct us in this process (apart from the small section in the wikitech Prometheus docs)? Thank you for reaching out! In the case of metrics generated from within Mediawiki you are correct that we're not there yet for off the shelf Mediawiki Prometheus metrics in production (though cc @colewhite and @Krinkle as I would love to be wrong here!). For non-Mediawiki metrics though (and new projects) the recommendation is to steer away from Statsd/Graphite and use Prometheus instead. My understanding is that you are solving for the former problem (i.e. MW) (?) TASK DETAIL https://phabricator.wikimedia.org/T297494 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ItamarWMDE, fgiunchedi Cc: Krinkle, colewhite, fgiunchedi, Aklapper, Manuel, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T297145: Ask for regular backups of our Wikidata Graphite data
fgiunchedi added a comment. This is complete I believe, we're backing up the `daily` hierarchy now TASK DETAIL https://phabricator.wikimedia.org/T297145 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Lucas_Werkmeister_WMDE, Manuel, Aklapper, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi closed this task as "Resolved". fgiunchedi claimed this task. fgiunchedi added a comment. I'm tentatively resolving the task since all short term mitigations are completed, feel free to reopen if sth is amiss TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. In T294355#7563057 <https://phabricator.wikimedia.org/T294355#7563057>, @Manuel wrote: > Thank you for the suggestion @fgiunchedi! Do we have an explanation somewhere of how to do this? Sure no problem! My understanding is that these metrics are published/pushed somewhat infrequently by background jobs, therefore a good starting point would be https://wikitech.wikimedia.org/wiki/Prometheus#Ephemeral_jobs_(Pushgateway) . Happy to provide more guidance/info on T297494 <https://phabricator.wikimedia.org/T297494> as well though TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. @Manuel @Lydia_Pintscher going forward I suggest also investing resources to switch to Prometheus as the supported metric system. Graphite is deprecated and in "life support" mode while all producers (essentially mediawiki and related) are being ported over, thanks! TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. In T294355#7559074 <https://phabricator.wikimedia.org/T294355#7559074>, @Lucas_Werkmeister_WMDE wrote: > In T294355#7531241 <https://phabricator.wikimedia.org/T294355#7531241>, @fgiunchedi wrote: > >> In T294355#7531236 <https://phabricator.wikimedia.org/T294355#7531236>, @Lucas_Werkmeister_WMDE wrote: >> >>> I’m not sure I understand the discussion correctly :) do you still need a list of paths to back up, or does it look like we can back up everything now? >> >> What's "everything" in this context? :) If you are talking about `daily` then yes it does look like it! > > I was thinking of everything, even non-daily stuff, but it looks like `daily` would actually be enough for us. Manuel created a list of important dashboards in T297145 <https://phabricator.wikimedia.org/T297145>; the topics they use are: Ok! Please see the related review to start backing up `daily`. I've added @jcrespo too for signing off purposes TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. In T294355#7531236 <https://phabricator.wikimedia.org/T294355#7531236>, @Lucas_Werkmeister_WMDE wrote: > I’m not sure I understand the discussion correctly :) do you still need a list of paths to back up, or does it look like we can back up everything now? What's "everything" in this context? :) If you are talking about `daily` then yes it does look like it! TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. In T294355#7528880 <https://phabricator.wikimedia.org/T294355#7528880>, @jcrespo wrote: > One more question, to finally decide if setting up weekly full backups or daily but incremental- do all files mostly change completely, or only a subset of them? Incrementals are able to be done with file granularity only (it will backup fully files as long as its path or hash has changed), if value.wsp changes every minute, and there is only 1 per value, we will do "weekly only full", otherwise the daily incrementals may be preferred. My expectation is that most files we're backing up will change (otherwise it means the metric files are not being updated, which would make the backups less relevant) so definitely +1 for e.g. a weekly full backup > If we end up doing weekly fulls, 11GB * 12 weeks of retention = 130 GB, which we can handle with no issue. Thank you that's good to know! TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. In T294355#7527157 <https://phabricator.wikimedia.org/T294355#7527157>, @jcrespo wrote: > number of files are (within reason) a non-blocker for bacula, as files are packaged into volumes. It is true that each file is stored as a mysql record, but that should be able to scale until dozens of (US) billons, although it may be slow to recover when rebuilding metadata. > > Most limiting factor would be the overall size + backup frequency for capacity planning. We don't have a lot of temporal data backed up, so not sure if we could come up with a strategy that saves space (e.g. if data is immutable, we may want to avoid full backups every day). What is the file/directory structure? If data is below e.g. 100GB I would consider it "small" and not requiring optimization. > > The typical backup schedule is incrementals of a set of paths every day, differentials every fortnite, and fulls monthly- however it is highly customizable per job. Thank you that's helpful to know, my hunch is that we'd want every other week backups since this is mainly a safety measure. File structure is one file per metric for graphite, with the filesystem path mirroring the graphite path (e.g. `foo.bar.baz.value` will be `/foo/bar/baz/value.wsp` on the filesystem). All files are expected to be around ~100k (e.g. the `daily` top level directory I mentioned earlier is ~100k files and 11G in size). TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a subscriber: jcrespo. fgiunchedi added a comment. In T294355#7527026 <https://phabricator.wikimedia.org/T294355#7527026>, @Lucas_Werkmeister_WMDE wrote: > Sounds like a good idea to me, I can’t judge how much would fit in Bacula. Do you need a list of important metrics (worth backing up)? Yes a list of "paths" in the metrics hierarchy would be greatly helpful. re: acceptable limits of files for bacula jobs I'm looping in @jcrespo for guidance/assistance TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. I've sent the incident up for review, what do you think re: my proposal of adding parts of the hierarchy to bacula (if it is feasible in terms of number of files, e.g. `daily` is ~100k files now) TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. Draft incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-10-29_graphite Please feel free to integrate/change as needed. I'll be OOO until the 18th and I'll pick this back up TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. Audit completed, what I did is count the number of null data points in the year leading to the graphite2003 reimage (i.e. the first reimage, where the backfill would have first failed) from 2020/10/14 to 2021/10/11 (first column). And the number of nulls after the first reimage (from 2021/10/12 to 2021/10/20) in the second column. The files for which backfill failed would have a high number of nulls in the first column but low in the second (i.e. datapoints are being appended now, but haven't for the last year). The full list for metrics with more than 10 nulls in last year but less than 10 in the last week is at https://people.wikimedia.org/~filippo/nulls-T294355 (12MB file). I believe these were all the metrics affected by the failed backfill and for which we lost historical data. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. Status update: I'm running a full audit on all ~4M metric files looking for similar cases. The backfill from yesterday completed in the mean time and some metrics were able to be backfilled successfully. I'll be following up with an incident report about this -- again my apologies for the unexpected data loss during migration and backfill. In terms of action items: we currently don't backup graphite metric files, mostly due to the sheer number of files and space they take. However if a subset of metric files in a directory hierarchy isn't too big (I don't have the exact number on hand for "big" but I'd say low tens of thousands, to be confirmed) then it should be doable to back up in bacula. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
fgiunchedi added a comment. Status update: the backfill is still ongoing since I lowered the concurrency. The good news is that some metrics are already backfilled, e.g. api backend summary: https://grafana.wikimedia.org/d/2/api-backend-summary?viewPanel=31=1=161723520=163511000 The bad news is that I suspect the first backfill on Oct 11th (i.e. when we reimaged and then backfilled graphite2003) suffered from the same undetected problem, therefore in that case we do have data loss unfortunately TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2022
fgiunchedi added a comment. @Lucas_Werkmeister_WMDE thank you for the report. Yes pretty sure the graphite bullseye migration is related. We backfilled graphite1004 from graphite2003 (which in turn was the first host we reimaged, and backfilled it from graphite1004), I suspect some metric files did backfill fully and some others didn't (I don't know why exactly yet). I verified taking one of your examples for mediawiki API `/srv/carbon/whisper/MediaWiki/api/query/executeTiming/sample_rate.wsp` has historical data on graphite2003 but not graphite1004 (as experienced). So I think what's needed is to run a backfill again (on the metrics that we know are missing data first), this is a safe operation because data gets merged. I'll try that tomorrow and report back my findings. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Invadibot, maantietaja, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294014: Invalid wikidata daily metrics received
fgiunchedi added a project: Observability-Metrics. TASK DETAIL https://phabricator.wikimedia.org/T294014 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Aklapper, fgiunchedi, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294014: Invalid wikidata daily metrics received
fgiunchedi created this task. fgiunchedi added a project: Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Similarly to T293329: Invalid wikidata graphite metrics received <https://phabricator.wikimedia.org/T293329> there are `wikidata.daily` metrics where the value itself is missing, and thus can't be ingested: 21/10/2021 03:00:00 :: invalid line received from client 127.0.0.1:53686, ignoring [daily.wikidata.social.irc.members 1634785200] 21/10/2021 03:00:00 :: invalid line received from client 127.0.0.1:33234, ignoring [daily.wikidata.social.twitter.followers 1634785200] 21/10/2021 03:00:01 :: invalid line received from client 127.0.0.1:41170, ignoring [daily.wikidata.social.facebook.likes 1634785200] TASK DETAIL https://phabricator.wikimedia.org/T294014 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Aklapper, fgiunchedi, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293329: Invalid wikidata graphite metrics received
fgiunchedi added a comment. @Lucas_Werkmeister_WMDE can confirm the invalid metrics don't show up anymore, thank you! I found others for `wikidata.daily` but will file a separate task for that: carbon-cache@b/listener.log:21/10/2021 03:00:00 :: invalid line received from client 127.0.0.1:53686, ignoring [daily.wikidata.social.irc.members 1634785200] carbon-cache@d/listener.log:21/10/2021 03:00:00 :: invalid line received from client 127.0.0.1:33234, ignoring [daily.wikidata.social.twitter.followers 1634785200] carbon-cache@e/listener.log:21/10/2021 03:00:01 :: invalid line received from client 127.0.0.1:41170, ignoring [daily.wikidata.social.facebook.likes 1634785200] The task is resolved on my end, not sure if there's anything else you need to wrap up? TASK DETAIL https://phabricator.wikimedia.org/T293329 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE, fgiunchedi Cc: Lucas_Werkmeister_WMDE, Michael, fgiunchedi, Addshore, Aklapper, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293329: Invalid wikidata graphite metrics received
fgiunchedi created this task. fgiunchedi added projects: Wikidata, Observability-Metrics. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION I noticed this in carbon logs on graphite1004, looks like some wikidata processes don't send a metric value ==> carbon-cache@b/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.freshest.lag 1634198220) received from client 127.0.0.1:40438, ignoring ==> carbon-cache@c/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.freshest.pending 1634198220) received from client 127.0.0.1:49942, ignoring 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.stalest.pending 1634198220) received from client 127.0.0.1:49942, ignoring ==> carbon-cache@d/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.median.pending 1634198220) received from client 127.0.0.1:38194, ignoring ==> carbon-cache@a/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.median.lag 1634198220) received from client 127.0.0.1:43602, ignoring ==> carbon-cache@b/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.average.lag 1634198220) received from client 127.0.0.1:40438, ignoring ==> carbon-cache@d/listener.log <== 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.stalest.lag 1634198220) received from client 127.0.0.1:38194, ignoring 14/10/2021 07:57:00 :: invalid line (wikidata.dispatch.average.pending 1634198220) received from client 127.0.0.1:38194, ignoring cc @Addshore TASK DETAIL https://phabricator.wikimedia.org/T293329 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Addshore, Aklapper, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: lmata, fgiunchedi Cc: fgiunchedi, Aklapper, Suran38, Biggs657, Invadibot, Lalamarie69, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Kotchchanipa, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290080: Move wikidata lag checks off Icinga
fgiunchedi added a comment. I agree, this is done TASK DETAIL https://phabricator.wikimedia.org/T290080 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, fgiunchedi Cc: colewhite, Addshore, Ladsgroup, fgiunchedi, Aklapper, the0001, Invadibot, Zabe, Selby, AndreCstr, maantietaja, XeroS_SkalibuR, lmata, Hazizibinmahdi, Akuckartz, Iflorez, alaa_wmde, DannyS712, Nandana, Mirahamira, Lahi, Gq86, herron, Markhalsey, GoranSMilovanovic, Jayprakash12345, Chicocvenancio, QZanden, LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290080: Move wikidata lag checks off Icinga
fgiunchedi added a comment. Thank you @Addshore and @Ladsgroup ! Much easier to go Grafana for now, I've retitled/repurposed the task and thanks for your help on T240685 <https://phabricator.wikimedia.org/T240685> ! TASK DETAIL https://phabricator.wikimedia.org/T290080 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, fgiunchedi Cc: colewhite, Addshore, Ladsgroup, fgiunchedi, Aklapper, Suran38, Biggs657, the0001, Invadibot, Lalamarie69, Zabe, Selby, AndreCstr, maantietaja, XeroS_SkalibuR, lmata, Juan90264, Alter-paule, Hazizibinmahdi, Beast1978, Un1tY, Akuckartz, Hook696, Iflorez, Kent7301, alaa_wmde, joker88john, DannyS712, CucyNoiD, Nandana, Mirahamira, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, Markhalsey, GoranSMilovanovic, Jayprakash12345, Chicocvenancio, QZanden, LawExplorer, Lewizho99, Volans, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290080: Move wikidata lag checks off Icinga
fgiunchedi renamed this task from "Collect wikidata/siteinfo in Prometheus" to "Move wikidata lag checks off Icinga". fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T290080 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, fgiunchedi Cc: colewhite, Addshore, Ladsgroup, fgiunchedi, Aklapper, Suran38, Biggs657, the0001, Invadibot, Lalamarie69, Zabe, Selby, AndreCstr, maantietaja, XeroS_SkalibuR, lmata, Juan90264, Alter-paule, Hazizibinmahdi, Beast1978, Un1tY, Akuckartz, Hook696, Iflorez, Kent7301, alaa_wmde, joker88john, DannyS712, CucyNoiD, Nandana, Mirahamira, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, Markhalsey, GoranSMilovanovic, Jayprakash12345, Chicocvenancio, QZanden, LawExplorer, Lewizho99, Volans, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T290080: Collect wikidata/siteinfo in Prometheus
fgiunchedi created this task. fgiunchedi added projects: observability, MediaWiki-General, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION This is a followup in the context of migrating wikidata alerts to AlertManager (T287741 <https://phabricator.wikimedia.org/T287741>). The remaining Icinga checks are mostly about wikidata lag, namely fetching `https://www.wikidata.org//w/api.php?action=query=siteinfo=json=statistics` to compare the lag value to a threshold (in a shell script) The `siteinfo` statistics are of general interest and I think it makes sense to have them in Prometheus. In practical terms we'd teach production Prometheus to fetch the siteinfo api call like any other endpoint. A practical but fictional example of what I mean is the following: $ curl -s 'https://www.wikidata.org/w/api.php?action=query=siteinfo=PROMETHEUS=statistics' mediawiki_api_dispatch_median_lag_seconds: 1 mediawiki_api_dispatch_freshest_lag_seconds: 1 mediawiki_api_dispatch_stalest_lag_seconds: 1 ... The exact metric names and labels are TBD but that's the general idea. This isn't full Prometheus support (in place of statsd, see T240685 <https://phabricator.wikimedia.org/T240685>) but IMHO going in the right direction, also in the context of T249164 <https://phabricator.wikimedia.org/T249164>. What do you think? TASK DETAIL https://phabricator.wikimedia.org/T290080 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Ladsgroup, fgiunchedi, Aklapper, the0001, Invadibot, Zabe, Selby, AndreCstr, maantietaja, XeroS_SkalibuR, lmata, Akuckartz, DannyS712, Nandana, Mirahamira, Lahi, Gq86, herron, Markhalsey, GoranSMilovanovic, Jayprakash12345, Chicocvenancio, QZanden, LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: lmata, fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Kotchchanipa, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from Icinga to alert manager exclusively. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281454: Onboard teams with Prometheus-based alerts to AlertManager
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from Icinga to alert manager exclusively. TASK DETAIL https://phabricator.wikimedia.org/T281454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Jdlrobson, fgiunchedi, Aklapper, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T287741: Convert wikidata-alerts grafana dashboard to AlertManager
fgiunchedi added a parent task: T281359: Onboard teams with Grafana alerts to AlertManager. TASK DETAIL https://phabricator.wikimedia.org/T287741 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Aklapper, Addshore, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi added a subtask: T287741: Convert wikidata-alerts grafana dashboard to AlertManager. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi closed subtask T282806: Port traffic/netops grafana alerts to AlertManager as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281359: Onboard teams with Grafana alerts to AlertManager
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T281359 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Invadibot, MPhamWMF, maantietaja, lmata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T272128: Fix tracking for query service UI
fgiunchedi added a comment. In T272128#7212524 <https://phabricator.wikimedia.org/T272128#7212524>, @Ladsgroup wrote: > Tomorrow all metrics starting with `wikibase.queryService.ui.app.` should be migrated to `wikibase.queryService.ui.index.app.` I will deploy it around 8:30 UTC For my reference at deploy time on the graphite hosts this translates to: sudo -u _graphite -s /bin/bash install -d /srv/carbon/whisper/wikibase/queryService/ui/index cp -v /srv/carbon/whisper/wikibase/queryService/ui/app /srv/carbon/whisper/wikibase/queryService/ui/index We'll be copying instead of moving in case sth goes wrong we'd have the old data at least, old metrics can be then reclaimed/deleted TASK DETAIL https://phabricator.wikimedia.org/T272128 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, fgiunchedi Cc: fgiunchedi, Lucas_Werkmeister_WMDE, Manuel, Addshore, Ladsgroup, Aklapper, Lydia_Pintscher, Invadibot, MPhamWMF, maantietaja, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, alaa_wmde, Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, Mahir256, QZanden, EBjune, merbst, LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T262741: "Wikidata API format usage" Grafana dashboard is empty
fgiunchedi added a project: observability. TASK DETAIL https://phabricator.wikimedia.org/T262741 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Lydia_Pintscher, Addshore, abian, Aklapper, Invadibot, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Imarlier, Lahi, Gq86, herron, GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T274249: Offboard wdqs-admins from legacy pager in Icinga
fgiunchedi closed this task as "Resolved". fgiunchedi claimed this task. fgiunchedi added a comment. Chatted with @gehel and concluded we're ok to offboard him and @RKemper from legacy paging as-is! TASK DETAIL https://phabricator.wikimedia.org/T274249 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Gehel, RKemper, Aklapper, MPhamWMF, lmata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
fgiunchedi added a comment. Restricted Application added a project: wdwb-tech-focus. random-ish update re: checkpoint storage after a chat with @Zbyszko: the current situation is that we're using thanos-swift cluster for wdqs flink checkpoints. This is meant to be a temporary allocation and wdqs to be eventually moved off thanos-swift cluster. Things have shifted a bit and we're building MOSS (misc object storage service) as a separate swift cluster exactly for these use cases (and more, T264291 <https://phabricator.wikimedia.org/T264291>). The plan is thus to keep using thanos-swift for the time being until moss is online and then migrate the updater there. TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Nuria, Addshore, Milimetric, JAllemandou, Ottomata, Pchelolo, Joe, Aklapper, dcausse, Zbyszko, Gehel, Ramtin0071, MPhamWMF, Devnull, lmata, Muchiri124, CBogen, Akuckartz, 4748kitoko, Legado_Shulgin, Nandana, Namenlos314, Akovalyov, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, abian, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3
fgiunchedi added a project: observability. TASK DETAIL https://phabricator.wikimedia.org/T269204 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a comment. In T246004#6567338 <https://phabricator.wikimedia.org/T246004#6567338>, @Zbyszko wrote: > Thank you all for swift (pun intended) action! haha! the account is setup now, I've written the credentials in your home on `deploy1001` TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, Alter-paule, NavinRizwi, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a comment. In T246004#6560303 <https://phabricator.wikimedia.org/T246004#6560303>, @Ottomata wrote: > I don't know! @fgiunchedi how does one access the cluster? @elukey can check the network VLAN ACLs and update accordingly. The canonical url is https://thanos-swift.discovery.wmnet TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a comment. Ok, thank you for the information. It doesn't seem we have an isolated test environment anyways so even though I'm reluctant we'll have to test on the production swift cluster. A middle ground I suppose would be to create the accounts on the `thanos` swift cluster first, which is functionally the same as production but not in the hot-path for serving content. Let me know how you'd like to proceed! TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265015: TypeError: $.widget is not a function -- User script error
fgiunchedi removed a project: Wikimedia-Logstash. fgiunchedi added a comment. Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case! TASK DETAIL https://phabricator.wikimedia.org/T265015 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Michael, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265035: Gadget Error: "Uncaught TypeError: $(...).css(...).draggable is not a function"
fgiunchedi removed a project: Wikimedia-Logstash. fgiunchedi added a comment. Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case! TASK DETAIL https://phabricator.wikimedia.org/T265035 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Magnus, Aklapper, Michael, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265037: User Script error: "ReferenceError: wikibase is not defined"
fgiunchedi removed a project: Wikimedia-Logstash. fgiunchedi added a comment. Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case! TASK DETAIL https://phabricator.wikimedia.org/T265037 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, matej_suchanek, Aklapper, Michael, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265022: Client (Gadget?) Error: "NS_ERROR_FILE_CORRUPTED: "
fgiunchedi removed a project: Wikimedia-Logstash. fgiunchedi added a comment. Removing `wikimedia-logstash` since it doesn't seem to be a logstash-specific issue, please add back if that's not the case! TASK DETAIL https://phabricator.wikimedia.org/T265022 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Michael, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a comment. In T246004#6520077 <https://phabricator.wikimedia.org/T246004#6520077>, @Zbyszko wrote: > @fgiunchedi Currently, Flink pipeline resides on the Analytics Hadoop cluster. As for the question whether Flink creates it's containers - I think not, it did complain when there was no container, so I assume it expects one. Ack, thank you! For a POC / test I'd still prefer production resources not to be used, especially as we don't know (I think?) what the write patterns are like. At the same time it doesn't look like running docker on stat hosts is a thing (?) (cc @ottomata @elukey ?). Do you have a sense (e.g. order of magnitude) of how many writes we're talking about (for the test/POC and also full scale), at what concurrency and how much data would be written at a time ? That'd help greatly with understanding if we can go ahead with the production Swift accounts. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a comment. In T246004#6501106 <https://phabricator.wikimedia.org/T246004#6501106>, @Zbyszko wrote: > @fgiunchedi We estimate we'd need around 500GB of storage for the streaming updater (not accounting for replicas). Our use case is almost always write only (checkpoints are read only on pipeline restarts, which ideally will be done rarely) - but we have a elasticity when it comes to configuration of the checkpoints interval. 500G seems reasonable to me, do you have a sense how/if this size increases over time? (need to know in general but not a blocker ATM). Do you know if flink takes care of creating the containers too once it has access to swift? I'm asking because on container creation we can pick whether data will be stored on hdd or ssd. In T246004#6501794 <https://phabricator.wikimedia.org/T246004#6501794>, @Zbyszko wrote: > @fgiunchedi unfortunately, there is no docker on stat instances so I'm unable to test swift that way. I'd still prefer to have some container on a already running service (whichever is accessible from analytics cluster). Test we want to set up would involve longer running service, starting from longer intervals, going back to shorter ones. We want to set up full end to end solution, so it'd make sense to use a stable solution anyway. I understand where you are coming from and wanting to setup a full end to end solution. Where is (local?) testing of the Flink/WDQS pipeline happening at the moment? I'm asking because wherever that environment is then it'd be good to have a (even minimal, single host) Swift cluster to test the integration. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
fgiunchedi added a subscriber: JMeybohm. fgiunchedi added a comment. @Zbyszko re: docker and swift. @JMeybohm suggested using https://github.com/swiftstack/docker-swift (and possibly lowering auth token TTLs to make sure renewing expired tokens works as expected) re: monitoring, we have production dashboards at https://grafana.wikimedia.org/d/OPgmB1Eiz/swift. I'm not sure about monitoring docker-swift, at a very basic level though swift will send statsd metrics out, so **temporarily** you can send to `statsd.eqiad.wmnet` for testing purposes. HTH! TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T238540: Delete grafana dashboard, https://grafana.wikimedia.org/d/000000599/wikibase-wb_terms-newitemidformatter
fgiunchedi edited projects, added Traffic, observability; removed Graphite. Restricted Application added a project: Operations. TASK DETAIL https://phabricator.wikimedia.org/T238540 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Addshore, darthmon_wmde, Legado_Shulgin, DannyS712, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, Volans, _jensen, rosalieper, Scott_WUaS, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, Robin.guo, Imarlier ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T238540: Delete grafana dashboard, https://grafana.wikimedia.org/d/000000599/wikibase-wb_terms-newitemidformatter
fgiunchedi added a comment. I can confirm that a DELETE of https://grafana.wikimedia.org/api/dashboards/uid/00599 results in a 403, further I don't see the request reaching grafana1001's apache logs. I'm adding #traffic <https://phabricator.wikimedia.org/tag/traffic/> since this looks like a regression, perhaps ATS is involved. TASK DETAIL https://phabricator.wikimedia.org/T238540 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, Aklapper, Addshore, darthmon_wmde, DannyS712, Nandana, Robin.guo, Imarlier, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T204364: Rate limit wdqs logs
fgiunchedi added a project: observability. TASK DETAIL https://phabricator.wikimedia.org/T204364 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel, fgiunchedi Cc: gerritbot, Smalyshev, fgiunchedi, Gehel, Aklapper, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, Legado_Shulgin, DannyS712, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Zppix, Volans, Maathavan, _jensen, rosalieper, Cirdan, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, jeremyb, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T136852: Wikibase\Client\Changes\WikiPageUpdater logging is very verbose
fgiunchedi added a project: observability. TASK DETAIL https://phabricator.wikimedia.org/T136852 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bd808, fgiunchedi Cc: gerritbot, hoo, Addshore, Aklapper, bd808, Zppix, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Volans, Maathavan, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331, fgiunchedi, jeremyb, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T178530: Improve field mapping for nginx logstash
fgiunchedi added a project: observability. TASK DETAIL https://phabricator.wikimedia.org/T178530 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: debt, fgiunchedi Cc: Stashbot, gerritbot, dcausse, Gehel, EBernhardson, Aklapper, Smalyshev, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Volans, Maathavan, _jensen, rosalieper, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, fgiunchedi, jeremyb, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T221774: Add Wikidata query service lag to Wikidata maxlag
fgiunchedi added a comment. In T221774#5155621 <https://phabricator.wikimedia.org/T221774#5155621>, @hoo wrote: > Possible way to do this: > > Create `PrometheusBlazegraphLagService` class which internally fetches the lag from a given Blazegraph instance like `curl "http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=scalar(time()%20-%20blazegraph_lastupdated%7Binstance%3D%22wdqs1005.eqiad.wmnet%3A9193%22%7D)"` (where `wdqs1005.eqiad.wmnet` is to be replaced by the hostname). That would be cached (given we don't want to hit Prometheus often and as we care for lag in the 30-60m range, fetching this once ever 1-5m should be fine… this could maybe even be done in a Job). We would do that for all known `wdqs` instances and then sum/average/… the results. > > This value would then be used for adjusting maxlag, as described above. > > Things to consider: > > - Is going to Prometheus directly the right thing to do? > - How often can we sanely hit Prometheus? > - Where do we want to manage the list of WDQS instance for this? (Or I guess can we also ask Prometheus for all metrics at once?) re: frequency even once a minute would be fine, since the query isn't heavy to run, and yes you can ask about all instances at once, or e.g. take the `max()`. Which leads me to a question re: servers in maintenance, where / how is the list maintained or will be maintained of all instances and/or instances in maintenance? I'm asking because if the list of instances that should be queried is known anyways IMHO it'd be simpler to query the lag via sparql and keep the prometheus out of the loop entirely. I'm saying this because IIRC the "lastupdated" value would go blazegraph -> prometheus -> mediawiki TASK DETAIL https://phabricator.wikimedia.org/T221774 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, fgiunchedi Cc: Ladsgroup, Smalyshev, fgiunchedi, hoo, Daniel_Mietchen, MisterSynergy, Addshore, Sjoerddebruin, Aklapper, Lucas_Werkmeister_WMDE, darthmon_wmde, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T187960: Rack/cable/configure asw2-a-eqiad switch stack
fgiunchedi updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, fgiunchedi Cc: akosiaris, Joe, fgiunchedi, hashar, Krinkle, ArielGlenn, jijiki, Addshore, MMiller_WMF, Catrope, elukey, Marostegui, Stashbot, Paladox, gerritbot, Aklapper, BBlack, Cmjohnson, ayounsi, alaa_wmde, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, kostajh, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Soum213, Taiwania_Justo, Thibaut120094, Wong128hk, Wikidata-bugs, aude, Southparkfan, mark, Lydia_Pintscher, Darkdadaah, faidon, Nikerabbit, Arrbee, santhosh, KartikMistry, Jdforrester-WMF, Mbch331, Jay8g, Ltrlg ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T208215: Metrics from wdqs updater JMX should be prefixed
fgiunchedi added a comment. Any update?TASK DETAILhttps://phabricator.wikimedia.org/T208215EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Aklapper, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, D3r1ck01, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T208215: Metrics from wdqs updater JMX should be prefixed
fgiunchedi created this task.fgiunchedi added a project: Wikidata-Query-Service.Restricted Application added a subscriber: Aklapper.Restricted Application added a project: Wikidata. TASK DESCRIPTIONNoticed this while investigating something else, metrics exposed by jmx_exporter running on wdqs-updater should be prefixed with wdqs_updater_ to make it more clear what they are referring to. I believe this can be achieved either via jmx_exporter config via rules or by Prometheus server itself at scrape time, with the former being preferred IMO. wdqs1004:~$ curl -s localhost:9101/metrics | grep -v ^# | grep metrics metrics_rdf_fetch_timer_98thPercentile 153.172966 metrics_constraint_fetch_timer_StdDev 13.365816886763675 metrics_kafka_changes_timer_Count 93957.0 metrics_rdf_fetch_timer_Min 78.936681 metrics_kafka_changes_timer_MeanRate 0.2910079145414642 metrics_rdf_fetch_timer_75thPercentile 113.298346TASK DETAILhttps://phabricator.wikimedia.org/T208215EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Aklapper, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T195121: Contribution from the IGN to Structured Data on Commons
fgiunchedi added a comment. In T195121#4529715, @aborrero wrote: In T195121#4527757, @fgiunchedi wrote: Sounds like a nice project! With my swift maintainer hat on, testing a single 200-300 GB chunk of data sounds good to me. Let's coordinate though before uploading the full data set because swift is pending its annual expansion (T201937) and I'd like to have that completed to not push swift disk usage too much with substantial uploads. Perhaps I didn't use correct words. Also, I don't know in deep how data looks like. But I believe files are small, like map tiles and other images. In this case was using data chunk to refer to the downloadable files that IGN offers, which seems to composed of many of these map tiles or other images and metadata. Indeed, I was referring to those downloadable chunks you mentioned. Also as a data point mediawiki isn't going to allow uploads for single files greater than 4-5GB (!)TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T195121: Contribution from the IGN to Structured Data on Commons
fgiunchedi added a comment. Sounds like a nice project! With my swift maintainer hat on, testing a single 200-300 GB chunk of data sounds good to me. Let's coordinate though before uploading the full data set because swift is pending its annual expansion (T201937) and I'd like to have that completed to not push swift disk usage too much with substantial uploads.TASK DETAILhttps://phabricator.wikimedia.org/T195121EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Reedy, bd808, Aklapper, aborrero, SandraF_WMF, Platonides, Rodelar, abian, stebsco, AndyTan, sietec, Zylc, 1978Gage2001, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, GoranSMilovanovic, Chicocvenancio, QZanden, Tbscho, Tramullas, Acer, LawExplorer, JJMC89, Susannaanas, srodlund, Luke081515, Aschroet, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Gryllida, Ricordisamoa, Lydia_Pintscher, Fabrice_Florin, Raymond, scfc, Steinsplitter, Mbch331, Krenair, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T195520: Multiple projects reporting Cannot access the database: No working replica DB server
fgiunchedi closed subtask T195530: status.wikimedia.org showing all lights green during major outage as "Invalid". TASK DETAILhttps://phabricator.wikimedia.org/T195520EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, fgiunchediCc: Aklapper, Jdforrester-WMF, Mateusz_Konieczny, eranroz, MichaelSchoenitzer_WMDE, Tractopelle-jaune, Florian, Jeff_G, Vachovec1, Tbayer, Ivanhercaz, Lixxx235, Stashbot, Addshore, MrFulano, daniel, Lydia_Pintscher, Tarawneh, PetrohsW, Sunpriat2, Ladsgroup, Legoktm, Thryduulf, matmarex, Krenair, dgstranz, Cirdan, Amorymeltzer, Yarl, Doc_James, Masti, Wiki-1776, Baloch007, abian, Pigsonthewing, putnik, Classicwiki, Jarekt, Jayprakash12345, 1339861mzb, KTC, Bharel, Daimona, Smartyllama, ToBeFree, Paladox, Urbanecm, Davey2010, alanajjar, Mh-3110, Lucas_Werkmeister_WMDE, JEumerus, Samtar, Lofhi, Marostegui, TerraCodes, Framawiki, Mainframe98, Stryn, Boshomi, Analytics.mediafiles, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, Jonas, Wong128hk, Wikidata-bugs, aude, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T192768: wdqs-updater crashing not cleanly
fgiunchedi added a comment. No planned upgrades ATM, though a newer upstream version might help with understanding (hopefully fixing) T192456: Prometheus metrics missing for some hosts too, so definitely welcome!TASK DETAILhttps://phabricator.wikimedia.org/T192768EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, Smalyshev, Aklapper, Gehel, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T186815: Badges not displaying on trwiki
fgiunchedi triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T186815EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Superyetkin, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T184434: prometheus-blazegraph-exporter failing to start after reboot
fgiunchedi closed this task as "Resolved".fgiunchedi claimed this task.fgiunchedi added a comment. Done, fix deployedTASK DETAILhttps://phabricator.wikimedia.org/T184434EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: fgiunchedi, gerritbot, MoritzMuehlenhoff, Aklapper, Muehlenhoff, Gehel, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T147328: Add http://tools.wmflabs.org/grafana-json-datasource as a datasource to production grafana instance
fgiunchedi added a project: User-fgiunchedi.fgiunchedi added a comment. @Addshore yes! I'll try taking a look in the next couple of weeks I thinkTASK DETAILhttps://phabricator.wikimedia.org/T147328EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, fgiunchediCc: fgiunchedi, gerritbot, StudiesWorld, Aklapper, Addshore, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Andrew-WMDE, Izno, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T160685: Increase $wgExpensiveParserFunctionLimit on nowiki
fgiunchedi triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T160685EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Lydia_Pintscher, Krinkle, Reedy, jeblad, Aklapper, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Salgo60, Zppix, Urbanecm, JEumerus, Volker_E, Tulsi_Bhagat, Izno, Luke081515, biplabanand, Wikidata-bugs, Snowolf, aude, GWicke, faidon, Matanya, Mbch331, Rxy, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T150356: Wikidata Query Service is overly verbose toward logstash
fgiunchedi triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T150356EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: fgiunchediCc: Smalyshev, Gehel, Aklapper, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147329: Add simple-json-datasource plugin to productrion grafana instance
fgiunchedi added a comment. Merged, though following up from IRC: we're keeping grafana labs/prod segregated in their datasources to avoid introducing more production/labs dependencies. It'd be nice if the tool ran somewhere in production. AFAICS it is stateless so that shouldn't be too hard to host.TASK DETAILhttps://phabricator.wikimedia.org/T147329EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, fgiunchediCc: fgiunchedi, gerritbot, yuvipanda, Aklapper, StudiesWorld, Addshore, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T133490: Wikidata Query Service REST endpoint returns truncated results
fgiunchedi triaged this task as "Normal" priority. TASK DETAIL https://phabricator.wikimedia.org/T133490 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Bovlb, Aklapper, Mushroom, Avner, debt, Gehel, D3r1ck01, FloNight, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002
fgiunchedi added a comment. drive-by comment: partman will likely to be adjusted too so we don't run into surprises when reprovisioning TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH, fgiunchedi Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T118850: Delete daily.wikidata.api.getclaims_property_use.* Graphite metrics
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
{{done}}
TASK DETAIL
https://phabricator.wikimedia.org/T118850
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Aklapper, Addshore, StudiesWorld, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T118836: Delete wikibase.dispatch.* metrics
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi closed this task as "Resolved".
fgiunchedi claimed this task.
fgiunchedi added a comment.
{{done}}
TASK DETAIL
https://phabricator.wikimedia.org/T118836
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: fgiunchedi
Cc: fgiunchedi, Addshore, Aklapper, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T119579: Additional diskspace of wdqs1001/wdqs1002
fgiunchedi triaged this task as "Normal" priority. fgiunchedi added a subscriber: fgiunchedi. TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
fgiunchedi added a subscriber: fgiunchedi. fgiunchedi added a comment. thanks for expanding on that, here's my (as the person who's been looking after our graphite stack) opinion: - graphite isn't really data warehouse, thus I wouldn't recommend it as the primary storage for the verbatim/authoritative data - though saving data in graphite for graphing/etc and archived elsewhere too I think would cater in this case - it is possible as @addshore suggests to not downsample daily data for a really long time, e.g. keeping a daily metric for e.g. 100y takes 438028 bytes on disk for each metric - an analytics graphite instance could help, it means maintenance of that too of course. - if the volume of metrics isn't very high (no idea on the order of magnitude though) then using the main graphite is certainly less overhead. To give an example, if we're talking about 10k distinct metrics that'd be no problem, 100k would be ATM. hope that helps! TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, fgiunchedi Cc: fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117732: Create a Graphite instance in the Analytics cluster
fgiunchedi added a comment. @addshore to clarify, more than functionality I was pointing out guarantees about the data stored. if the metrics are also being archived to hdfs for example so it is possible to dump/load into graphite then IMO that's acceptable. re: analytics graphite instance, I think there's value in a single shared instance for ease of use, even though for example grafana supports mixed dashboards so it is possible to collate multiple graphite sources TASK DETAIL https://phabricator.wikimedia.org/T117732 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: Lydia_Pintscher, fgiunchedi, Christopher, JanZerebecki, Nuria, Ottomata, Aklapper, Addshore, StudiesWorld, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117402: Enable retention of daily metrics for longer periods of time in Graphite
fgiunchedi added a subscriber: fgiunchedi. fgiunchedi added a comment. for long term data warehousing or analytics type of workflows using ourhadoop/analytics infrastructure will be more appropriate I think. graphite is more focused on operational metrics from applications, services and so on TASK DETAIL https://phabricator.wikimedia.org/T117402 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, fgiunchedi Cc: fgiunchedi, gerritbot, Addshore, Aklapper, Wikidata-bugs, aude ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Assigned] T95679: Make a puppet role that sets up a query service and loads it
fgiunchedi added a subscriber: fgiunchedi. fgiunchedi assigned this task to GLavagetto. fgiunchedi added a comment. moving to @joe TASK DETAIL https://phabricator.wikimedia.org/T95679 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GLavagetto, fgiunchedi Cc: fgiunchedi, Lydia_Pintscher, Matanya, gerritbot, Joe, Smalyshev, Liuxinyu970226, Aklapper, Manybubbles, Wikidata-bugs, RobH, aude, mark, faidon, scfc, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reassigned] T95679: Make a puppet role that sets up a query service and loads it
fgiunchedi reassigned this task from GLavagetto to Joe. fgiunchedi added a subscriber: GLavagetto. TASK DETAIL https://phabricator.wikimedia.org/T95679 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, fgiunchedi Cc: GLavagetto, fgiunchedi, Lydia_Pintscher, Matanya, gerritbot, Joe, Smalyshev, Liuxinyu970226, Aklapper, Manybubbles, Wikidata-bugs, RobH, aude, mark, faidon, scfc, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T84902: deploy haedus and capella with os for orientdb testing
fgiunchedi added a subscriber: fgiunchedi. fgiunchedi added a comment. looks like this is completed, anything else left @joe ? TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: fgiunchedi, RobH, Aklapper, Joe, aaron, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, GWicke, Manybubbles, daniel, mark, JanZerebecki, faidon, Dzahn, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
