[Wikidata-bugs] [Maniphest] T344027: Validation Error for eventlogging_WMDEBannerSizeIssue
Ottomata merged a task: T349702: eventlogging_WMDEBannerSizeIssue validation errors. Ottomata added subscribers: Ottomata, Ahoelzl, JAllemandou, tchin, gabriel-wmde, kai.nissen, gmodena. TASK DETAIL https://phabricator.wikimedia.org/T344027 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: gmodena, kai.nissen, gabriel-wmde, tchin, JAllemandou, Ahoelzl, Ottomata, Aklapper, KSarabia-WMF, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, GWicke, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342593: Five deleted Wikidata items pertaining to Wikimedia category pages still present in the Query Service
Ottomata added a comment. > I agree with @Milimetric here and we need to get a better sense of the quality of the EventBus/EventGate system T345195: [SPIKE] Can we identify indicators to inform an SLO for event emission and intake? <https://phabricator.wikimedia.org/T345195> TASK DETAIL https://phabricator.wikimedia.org/T342593 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, EBernhardson, ArielGlenn, Milimetric, dcausse, Nikki, Aklapper, Mahir256, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, BTullis, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, holger.knust, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T334558: [Analytics] Unique user-agents accessing Wikidata's REST API
Ottomata added a comment. ^ sounds good! `wikidata` could be an okay name too. I like functional groupings. However, that might overlap with some dags that the search platform team does for wikidata in their `search` instance. ¯\_(ツ)_/¯ TASK DETAIL https://phabricator.wikimedia.org/T334558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE, Ottomata Cc: mforns, xcollazo, Ottomata, lbowmaker, WMDE-leszek, AndrewTavis_WMDE, Michael, Manuel, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T334558: [Analytics] Unique user-agents accessing Wikidata's REST API
Ottomata added a comment. We should avoid team names in functional code / namespacing. Team names change often. However, wmde is more like 'wmf' in this case, and I think not likely to change. +1 to wmde. :) TASK DETAIL https://phabricator.wikimedia.org/T334558 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AndrewTavis_WMDE, Ottomata Cc: mforns, xcollazo, Ottomata, lbowmaker, WMDE-leszek, AndrewTavis_WMDE, Michael, Manuel, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Ottomata updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Ottomata added a subscriber: BTullis. Ottomata added a comment. > create a namespace for the rdf-streaming-updater on the dse-k8s cluster BTW, I _think_ there is more involved than just this Puppet patch? Reach out to @BTullis ? TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: BTullis, JMeybohm, gmodena, Ottomata, bking, Aklapper, dcausse, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Ottomata added a comment. > What namespace strategy should we use for flink jobs? A single one for all wmf flink jobs, per team, per project? One per application is what I'm expecting. TASK DETAIL https://phabricator.wikimedia.org/T326409 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, bking, Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T298305: Realtime editing UI and API
Ottomata added a comment. I'm not sure where this task belongs. What UI and API are you refering to? See also: https://stream.wikimedia.org/?doc https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams TASK DETAIL https://phabricator.wikimedia.org/T298305 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Naseweis520, Aklapper, Lectrician1, Astuthiodit_1, BTullis, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, holger.knust, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, GWicke, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T314835: wdqs space usage on thanos-swift
Ottomata added subscribers: dcausse, Ottomata.
Ottomata added a comment.
Hi, I don't know much about this, but I did a little bit of digging.
I can see that the flink session cluster jobmanager is taking checkpoints
every few seconds, for each of the jobs it is running:
[@deploy1002:/srv/deployment-charts/helmfile.d/services/rdf-streaming-updater]
(master+)[64d48331] ± kubectl logs -l component=jobmanager -c
flink-session-cluster-main -f
{"@timestamp":"2022-08-09T13:02:12.987Z", "log.level": "INFO",
"message":"Completed checkpoint 889862 for job ca62db1723c0dd82afa1fd846c9923bc
(447282465 bytes in 11509 ms).", "ecs.version":
"1.2.0","service.name":"main","event.dataset":"main.log","process.thread.name":"jobmanager-future-thread-1","log.logger":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator"}
{"@timestamp":"2022-08-09T13:02:21.485Z", "log.level": "INFO",
"message":"Triggering checkpoint 592244 (type=CHECKPOINT) @ 1660050141460 for
job 0cd81dd86866a8575cc187d92c98eb49.", "ecs.version":
"1.2.0","service.name":"main","event.dataset":"main.log","process.thread.name":"Checkpoint
Timer","log.logger":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator"}
...
I can also see that the taskmanagers seem to be re instantiating and
connecting a new kafka producer fairly frequently:
[@deploy1002:/srv/deployment-charts/helmfile.d/services/rdf-streaming-updater]
(master+)[64d48331] ± kubectl logs -l component=taskmanager -c
flink-session-cluster-main-taskmanager -f --since=48h --max-log-requests=6 | jq
.
However, I don't know if either of those behaviors is abnormal. Every few
seconds seems like a lot to take checkpoints, and I'd expect Kafka producers to
stay alive unless there is some problem, but I don't know for sure. We'll need
@dcausse here I think.
TASK DETAIL
https://phabricator.wikimedia.org/T314835
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Ottomata, dcausse, LSobanski, dr0ptp4kt, fgiunchedi, Aklapper, Raineydaz,
Astuthiodit_1, AWesterinen, MatthewVernon, BTullis, karapayneWMDE,
joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja,
Muchiri124, CBogen, ItamarWMDE, Akuckartz, Di3sel1975, Chambersjay, RhinosF1,
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Conradrock, Davinaclare77,
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0,
QZanden, EBjune, merbst, LawExplorer, Zppix, elukey, _jensen, rosalieper,
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude,
Tobias1984, Manybubbles, faidon, Mbch331, Jay8g
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams
Ottomata added a comment. (Oh, past me said this already... :p) TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams
Ottomata added a comment. > Here you must consume only one. Actually, this is curious. These are really distinct streams. We probably should have named them differently. We can't just pick eqiad.rdf-streaming-updater.mutation when the user connects to EventStreams in eqiad, because the next time around they might connect to EventStreams in codfw, depending on how they are routed. So, I think either we expose only one of these topics, OR, we expose them both but as different distinct streams, and the user has to pick one. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T296470: Initialize WCQS production servers
Ottomata added a comment. Ran on main-eqiad and main-codfw kafka: kafka topics --create --topic eqiad.mediainfo-streaming-updater.mutation --replication-factor 3 --partitions 1 kafka configs --alter --entity-type topics --entity-name eqiad.mediainfo-streaming-updater.mutation --add-config retention.ms=259200 kafka topics --create --topic codfw.mediainfo-streaming-updater.mutation --replication-factor 3 --partitions 1 kafka configs --alter --entity-type topics --entity-name codfw.mediainfo-streaming-updater.mutation --add-config retention.ms=259200 TASK DETAIL https://phabricator.wikimedia.org/T296470 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, Zbyszko, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application
Ottomata added a comment. ※\(^o^)/※ TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel, Ottomata Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher, VladimirAlexiev, karapayneWMDE, MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris, revi, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, Suran38, Invadibot, maantietaja, Peteosx1x, NavinRizwi, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata added a comment. 3 our of your 7 records in this hour have meta.id == null (including this one), and Refine deduplication is removing all but one of them. You should set a unique value for `meta.id` in your events, but, arguably, Refine should not consider events with null `meta.id` as duplicates. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata added a comment. INTEResting. Will investigate. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293195: Add MCR slot information to revision-create events
Ottomata added a comment. Done and deployed! TASK DETAIL https://phabricator.wikimedia.org/T293195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: odimitrijevic, Cparle, JAllemandou, Milimetric, Aklapper, Ottomata, Pchelolo, dcausse, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293195: Add MCR slot information to revision-create events
Ottomata added a comment. I was about to merge that today but then thought that your suggestion to ensure that properties validate with the additionalProperties stuff would be good to add first. So you could implement that :D :D TASK DETAIL https://phabricator.wikimedia.org/T293195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: odimitrijevic, Cparle, JAllemandou, Milimetric, Aklapper, Ottomata, Pchelolo, dcausse, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams
Ottomata added a comment. Oh, we'll also want to add create event schema and add it to the schema repo. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams
Ottomata added projects: Analytics, Event-Platform. Ottomata added a comment. > Here you must consume only one. Should we expose both? We'll need to declare these streams in EventStreammConfig, likely each as a distinct stream overriding the list of topics that make up the stream. Then we can edit EventStreams helmfile configs to expose those streams. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, RBrounley_WMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Nirmos, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293195: Add MCR slot information to revision-create events
Ottomata added subscribers: Milimetric, JAllemandou. Ottomata added projects: Analytics, Event-Platform, Data-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T293195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: JAllemandou, Milimetric, Aklapper, Ottomata, Pchelolo, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T276469: Cookbooks and / or operation procedures are adapted for the new Flink based WDQS Streaming Updater
Ottomata added a comment. Just making sure yall have considered using consumer offset sync as supported by MirrorMaker 2 <https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/>. Upgrading to MM2 would be a big deal, but we haven't done it because no one has needed it! If there is a need it would be a good reason to upgrade. TASK DETAIL https://phabricator.wikimedia.org/T276469 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Ottomata Cc: Ottomata, dcausse, Aklapper, Gehel, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T291089: Proposal: Generate Wikidata JSON & RDF dumps from Hadoop
Ottomata added a comment. > I imagine other sources like https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams would all have the same problems? Yes, EventStreams uses the same data. TASK DETAIL https://phabricator.wikimedia.org/T291089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, JAllemandou, Ladsgroup, dcausse, ArielGlenn, Ottomata, Addshore, Invadibot, maantietaja, jannee_e, Akuckartz, 4748kitoko, holger.knust, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T291089: Proposal: Generate Wikidata JSON & RDF dumps from Hadoop
Ottomata added a comment. > And the new query service flink updater could also make use of the RDF stream Perhaps the existing logic in the WDQS updater to generate its RDF stream could be factored out into its own service? Or, at least, it could emit its RDF stream as a side output into a Kafka topic? To be reliable for dumps, we'd have to fix T215001: Revisions missing from mediawiki_revision_create <https://phabricator.wikimedia.org/T215001> as you say, but I'm hoping we can do that within this FY. TASK DETAIL https://phabricator.wikimedia.org/T291089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, JAllemandou, Ladsgroup, dcausse, ArielGlenn, Ottomata, Addshore, Invadibot, maantietaja, jannee_e, Akuckartz, 4748kitoko, holger.knust, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T291089: Proposal: Generate Wikidata JSON & RDF dumps from Hadoop
Ottomata added a comment. > a reliable and consistent input (such as MediaWiki recent changes) I guess by this you mean polling the MW RecentChanges API? TASK DETAIL https://phabricator.wikimedia.org/T291089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, JAllemandou, Ladsgroup, dcausse, ArielGlenn, Ottomata, Addshore, Invadibot, maantietaja, jannee_e, Akuckartz, 4748kitoko, holger.knust, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, terrrydactyl, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T287641: Review wd_propertysuggester event logging stream config
Ottomata added a comment. Ok, if everything is working, you should be able to emit events in deployment-prep now. TASK DETAIL https://phabricator.wikimedia.org/T287641 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Martaannaj, Michaelcochez, Aklapper, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Sjoerddebruin, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T287641: Review wd_propertysuggester event logging stream config
Ottomata added a comment. Thanks, +1! Actually this in labs so should be no prob to just merge and deploy. Will do that for ya now. TASK DETAIL https://phabricator.wikimedia.org/T287641 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Martaannaj, Michaelcochez, Aklapper, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Sjoerddebruin, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T287641: Review wd_propertysuggester event logging stream config
Ottomata added a comment. Left some comments! :) TASK DETAIL https://phabricator.wikimedia.org/T287641 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Martaannaj, Michaelcochez, Aklapper, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Sjoerddebruin, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate
Ottomata closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, BTullis, Biggs657, odimitrijevic, Invadibot, Lalamarie69, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata closed subtask T273901: Automate event stream ingestion into HDFS for streams that dont use EventGate as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate
Ottomata added a project: Analytics-Kanban. Ottomata added a comment. Yes! We've done this now that we are using Gobblin instead of Camus. Moving this to our Kanban so we can ACK and close it this week. TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, BTullis, Biggs657, odimitrijevic, Invadibot, Lalamarie69, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage
Ottomata added a comment. @Addshore I've quickly read the task description but I have to admit I don't fully understand it yet (what gets stored where, etc.). Find me on IRC? or set up quick little meeting? :) TASK DETAIL https://phabricator.wikimedia.org/T214362 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T276595: Upgrade prometheus-jmx-exporter
Ottomata edited projects, added Analytics-Clusters; removed Analytics. TASK DETAIL https://phabricator.wikimedia.org/T276595 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, fgiunchedi, colewhite, Ramtin0071, MPhamWMF, Devnull, maantietaja, Wilmanbeno, lmata, Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Volans, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, JAllemandou, abian, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jayvdb, faidon, Addshore, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T276595: Upgrade prometheus-jmx-exporter
Ottomata added a comment. Hello! Does Analytics have to upgrade too? :) TASK DETAIL https://phabricator.wikimedia.org/T276595 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, fgiunchedi, colewhite, Ramtin0071, MPhamWMF, Devnull, maantietaja, Wilmanbeno, lmata, Muchiri124, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Volans, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, JAllemandou, abian, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jayvdb, faidon, Addshore, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. @Gehel @dcausse these events are now in HDFS. There aren't any Hive tables yet because no non-canary events have yet been ingested, and we filter out canary events from the Hive tables. TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate
Ottomata added a subscriber: Mholloway. Ottomata added a comment. Ok here's the idea: I add a new EventStreamConfig settings block called `consumers`, where we can add consumers by name, and then put in relevant settings for them. Those consumers would be responsible for using those settings. We could do the same for `producers` (or maybe just `producer` for single writer principal <https://www.confluent.io/blog/build-services-backbone-events/>). This would be relevant for things being discussed in T273235: Define event stream configuration syntax <https://phabricator.wikimedia.org/T273235> cc @Mholloway Example for consumers: Declare settings: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668124 Use them: https://gerrit.wikimedia.org/r/c/operations/puppet/+/668125 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate
Ottomata created this task. Ottomata added projects: Wikidata-Query-Service, Analytics, Event-Platform. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Currently, we configure Camus jobs to import events based on which eventgate cluster instance they come through. We do this because generally the throughput of each stream is relatively the same size for each eventgate cluster instance, which allows us to better tune the ingestion jobs. T269619: Create pipelines for late/spurious/failed events <https://phabricator.wikimedia.org/T269619> will create one of the first set of event platform based streams that does not flow through eventgate. We need to add a Camus job that knows how to import these kinda of events, probably by adding another stream config setting to indicate that they should be ingested by Camus. This may cause us to re-think which stream config setting we are using to configure ingestion altogether. TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, MPhamWMF, CBogen, Akuckartz, 4748kitoko, holger.knust, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T270371: wikimedia-event-utilities should provide tools for JVM based apps producing directly to kafka
Ottomata updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T270371 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, dcausse, Aklapper, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T270371: wikimedia-event-utilities should provide tools for JVM based apps producing directly to kafka
Ottomata added a project: Event-Platform. TASK DETAIL https://phabricator.wikimedia.org/T270371 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, dcausse, Aklapper, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. T270371 <https://phabricator.wikimedia.org/T270371> is great thank you! > If this kind of usecases is something we'd like to support We do. EventGate is a poor substitute for a full fledged Kafka client. Kafka has so many more features and knobs to dial, and for prod apps those are going to be important. TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. It depends on what you want to do :) EventGate will handle multi DC, filling some default values, and topic prefixes for you, but is an extra hop to Kafka. As a prod system in a language with a good Kafka client, producing to Kafka is totally allowed. You'd be the first main user of event platform not going through EventGate, but it is definitely something we want to support. Perhaps we'll want to build in some logic for doing some of the things EventGate is doing (as a proxy) into wikimedia-event-utilties (including validation, as you suggested). TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. @dcausse, will these be POSTed to an EventGate, or to produced directly to Kafka? TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. > would it make sense to add some helper functions to wikimedia-event-utilities for validating a json against its schema Sure that could be useful :) TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events
Ottomata added a comment. @dcausse for retrieving schemas, https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia-event-utilities/+/refs/heads/master might help. :) TASK DETAIL https://phabricator.wikimedia.org/T269619 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink
Ottomata added a subscriber: elukey. Ottomata added a comment. > apache bigtop (from cloudera hadoop). That includes flink debs, would we want to use that? OH HO! I did not know that. Hopefully that would be best! @elukey FYI TASK DETAIL https://phabricator.wikimedia.org/T266495 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265525: Wikibase\Client\Usage\Sql\SqlSubscriptionManager::subscribe: Expected mass rollback of all peer transactions (DBO_TRX set)
Ottomata removed a project: Analytics. Ottomata added a comment. Hm, is there a reason Analytics was tagged? Looks like a JobQueue error? Removing analytics, feel free to re-add if needed. TASK DETAIL https://phabricator.wikimedia.org/T265525 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, thcipriani, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Jonas, Verdy_p, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, 4748kitoko, Akovalyov, JAllemandou, terrrydactyl, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T265525: Wikibase\Client\Usage\Sql\SqlSubscriptionManager::subscribe: Expected mass rollback of all peer transactions (DBO_TRX set)
Ottomata added a project: Platform Engineering. TASK DETAIL https://phabricator.wikimedia.org/T265525 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Aklapper, thcipriani, Akuckartz, 4748kitoko, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, Nandana, Akovalyov, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Jonas, Verdy_p, JAllemandou, terrrydactyl, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Ottomata added a comment. Ok, it looks like we need to allow traffic to 10.2.2.54 and 10.2.1.54 from the Analytics VLAN. @elukey can you add that? TY! TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Ottomata Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Ottomata added a comment. I don't know! @fgiunchedi how does one access the cluster? @elukey can check the network VLAN ACLs and update accordingly. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, Ottomata Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink
Ottomata added a comment. Hm, I think we need to find a use case that can be done using Stream SQL repl. I don't think SRE will deploy a Java app e.g. during a DDoS. Can Flink's SQL REPL do something like https://www.confluent.io/stream-processing-cookbook/ksql-recipes/syslog-pattern-detection-alerting/ or even better https://www.confluent.io/stream-processing-cookbook/ksql-recipes/detecting-abnormal-transactions/ with windows? TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Aklapper, Ottomata, Gehel, dcausse, CDanis, Zbyszko, CBogen, Akuckartz, 4748kitoko, darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T255399: Prepare wdqs1009 to run the streaming updater
Ottomata added a comment. Done: kafka configs --alter --entity-type topics --entity-name wdqs_streaming_updater_test --add-config retention.ms=267840 TASK DETAIL https://phabricator.wikimedia.org/T255399 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, dcausse, Blissjay007, Oblanco79, Alter-paule, Beast1978, CBogen, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Namenlos314, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster
Ottomata added a comment. Did 31 days: $ kafka topics --alter --topic eqiad.mediawiki.revision-create --config retention.ms=267840 $ kafka topics --alter --topic codfw.mediawiki.revision-create --config retention.ms=267840 $ kafka topics --describe --topic '^(eqiad|codfw).mediawiki.revision-create$' Topic:codfw.mediawiki.revision-create PartitionCount:1 ReplicationFactor:3 Configs:retention.ms=267840 Topic: codfw.mediawiki.revision-create Partition: 0Leader: 1002 Replicas: 1002,1003,1004Isr: 1004,1002,1003 Topic:eqiad.mediawiki.revision-create PartitionCount:1 ReplicationFactor:3 Configs:retention.ms=267840 Topic: eqiad.mediawiki.revision-create Partition: 0Leader: 1003 Replicas: 1003,1002,1005Isr: 1002,1005,1003 TASK DETAIL https://phabricator.wikimedia.org/T253753 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Ottomata added a comment. While not a google doc, the parent ticket's description describes it pretty well: T244590: EPIC: Rework the WDQS updater as an event driven application <https://phabricator.wikimedia.org/T244590> TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Nuria, Addshore, Milimetric, JAllemandou, Ottomata, Pchelolo, Joe, Aklapper, dcausse, Zbyszko, Gehel, 4748kitoko, darthmon_wmde, Legado_Shulgin, Nandana, Akovalyov, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, terrrydactyl, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Ottomata added a comment. A nice feature of Flink is its support for both batch and stream processing. Ideally, we'd be able to build lambda architectures <https://en.wikipedia.org/wiki/Lambda_architecture> reusing most of the core data logic for streams and historical batch backfilling. Also python support and SQL querying on streams. :) TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Milimetric, JAllemandou, Ottomata, Pchelolo, Joe, Aklapper, dcausse, Zbyszko, Gehel, 4748kitoko, darthmon_wmde, Legado_Shulgin, Nandana, Akovalyov, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, terrrydactyl, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Ottomata added a project: Analytics. TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Milimetric, JAllemandou, Ottomata, Pchelolo, Joe, Aklapper, dcausse, Zbyszko, Gehel, 4748kitoko, darthmon_wmde, Legado_Shulgin, Nandana, Akovalyov, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, terrrydactyl, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater
Ottomata added a comment. Ping also @Pchelolo for comments on ^ TASK DETAIL https://phabricator.wikimedia.org/T247058 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Pchelolo, Joe, Aklapper, dcausse, Zbyszko, Gehel, darthmon_wmde, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T246004: Spike: Can/should Swift be used as Flink checkpoint backend?
Ottomata created this task. Ottomata added projects: Epic, Wikidata-Query-Service, Wikidata. TASK DESCRIPTION TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T244590: EPIC: Rework the WDQS updater as an event driven application
Ottomata added subscribers: JAllemandou, Ottomata. Ottomata added a comment. COOL! :) > it's important to note that the state of step 3 is tightly coupled with its dump and thus we will have to instantiate a new stream per imported dump. In other words a wdqs system imported using dump Y will have to consume the RDF stream generated from an initial state based on this same dump. This means that the RDF stream will be named against a particular dump instance. Hm. Would it be possible instead to lambda architecture this part? Instead of having to reload from a full dump and then recreate a new stream, could accomplish the same cleanups by backfilling from a batch job in Hadoop? I'm not sure I fully understand the 'cleanups' here. Are they not do-able with the stream because events representing some of the state changes don't exist (yet)? TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, Aklapper, Zbyszko, Gehel, dcausse, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T209655: Copy Wikidata dumps to HDFS
Ottomata renamed this task from "Copy Wikidata dumps to HDFs" to "Copy Wikidata dumps to HDFS". TASK DETAIL https://phabricator.wikimedia.org/T209655 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: WMDE-leszek, abian, leila, Ottomata, Nuria, GoranSMilovanovic, Addshore, JAllemandou, bmansurov, 4748kitoko, darthmon_wmde, DannyS712, Nandana, Akovalyov, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, terrrydactyl, Wikidata-bugs, aude, Capt_Swing, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure
Ottomata added a comment. > Sorry about this stupid queue name No worries, its beta ┐｜･ิω･ิ#｜┌ TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Igorkim78, JAllemandou, Ottomata, Smalyshev, Deskana, Aklapper, 4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, holger.knust, Meekrab2012, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Scott_WUaS, Cirdan, Jonas, FloNight, Xmlizer, mobrovac, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Assigned] T236500: large number of 504 errors from ulsfo
Ottomata assigned this task to ema. Ottomata added a comment. Ema looks like you are working on this. Assigning to you as part of clinic duty. Feel free to resolve if done. TASK DETAIL https://phabricator.wikimedia.org/T236500 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ema, Ottomata Cc: Ottomata, sbassett, CDanis, Vgutierrez, BBlack, Joe, Ladsgroup, ema, jijiki, Aklapper, Bugreporter, darthmon_wmde, JFishback_WMF, Dsharpe, Legado_Shulgin, DannyS712, Nandana, Amorymeltzer, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, HJiang-WMF, merbst, LawExplorer, Zppix, _jensen, rosalieper, Jonas, Xmlizer, dpatrick, Wong128hk, Luke081515, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Bawolff, Stype_and_Co.-WMF, Manybubbles, DerHexer, Southparkfan, Jalexander, Parent5446, Anomie, faidon, Grunny, MaxSem, csteipp, Mbch331, Rxy, Jay8g, fgiunchedi, Legoktm, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure
Ottomata added a comment. sparql/query schema is merged. We'll need to do an eventgate-analytics k8s deploy before it can be used. Let me know when you want to start testing this. We can also update eventgate-analytics in deployment-prep first if you can/want to test there. TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Igorkim78, JAllemandou, Ottomata, Smalyshev, Deskana, Aklapper, 4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, holger.knust, Meekrab2012, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Cirdan, Jonas, FloNight, Xmlizer, mobrovac, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure
Ottomata added a comment. > do we need to do something on the refinery/hadoop side to create the hive table Depending on the name of the stream, it will probably need whitelisted in a camus a refine job. > and/or add a purge mechanism for the 90days retention I believe the tables are (or should be) purged by default so if you just want the data dropped after 90 days it will be. TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Smalyshev, Deskana, Aklapper, 4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, holger.knust, Meekrab2012, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Cirdan, Jonas, FloNight, Xmlizer, JAllemandou, mobrovac, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T101013: Log Wikidata Query Service queries to the event gate infrastructure
Ottomata added a project: EventBus. Restricted Application added a project: Analytics. TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Smalyshev, Deskana, Aklapper, 4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, holger.knust, Meekrab2012, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Cirdan, Jonas, FloNight, Xmlizer, JAllemandou, mobrovac, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure
Ottomata added a comment. Q: I assume you'll want this to go into the eventgate-analytics instance, yes? I.e. You won't be building production services on this stream, just using it for offline analytics (well, offline performance testing). TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Ottomata, Smalyshev, Deskana, Aklapper, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, joker88john, ET4Eva, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Cirdan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888
Ottomata added a comment. Ah, hm ok. Actually, @elukey why can't we allow the VIP IP? We did this in T221690: Allow analytics VLAN to reach schema.svc.$site.wmnet <https://phabricator.wikimedia.org/T221690>, no? TASK DETAIL https://phabricator.wikimedia.org/T176875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: akosiaris, Ottomata, elukey, Smalyshev, Gehel, Addshore, Aklapper, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888
Ottomata added a comment. @Addshore, just saw T218710 <https://phabricator.wikimedia.org/T218710> and clicked through to here. If you use https://wikitech.wikimedia.org/wiki/HTTP_proxy, you can access wdqs.svc.eqiad.wmnet over HTTP from the analytics VLAN. TASK DETAIL https://phabricator.wikimedia.org/T176875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, elukey, Smalyshev, Gehel, Addshore, Aklapper, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, _jensen, rosalieper, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T225195: EventBus jobs failing heavily because of CentralNotice and WikibaseRepo
Ottomata added a subscriber: Umherirrender. Ottomata added a comment. @umherirrender I'd like to revert https://gerrit.wikimedia.org/r/c/integration/config/+/513663 as it is causing blockers for EventBus changes. Objections? TASK DETAIL https://phabricator.wikimedia.org/T225195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Umherirrender, Legoktm, Ottomata, AndyRussG, awight, thiemowmde, Aklapper, E.S.A-Sheild, darthmon_wmde, Premeditated, holger.knust, joker88john, CucyNoiD, Nandana, NebulousIris, Akovalyov, Tumz24, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, Jseddon, rosalieper, gabriel-wmde, DStrine, Wong128hk, JAllemandou, mobrovac, terrrydactyl, Wikidata-bugs, Snowolf, Base, aude, GWicke, Pcoombe, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T225195: EventBus jobs failing heavily because of CentralNotice and WikibaseRepo
Ottomata added a comment. The failures I see in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52212/console don't really have to do with nulls passed to EventBus though. They just look failing CentralNotice tests, possible due to bad fixture data? E.g. ... -'throttle' => 50 +'throttle' => 100 ... in CentralNotice/tests/phpunit/CentralNoticeTestFixtures.php TASK DETAIL https://phabricator.wikimedia.org/T225195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, AndyRussG, awight, thiemowmde, Aklapper, E.S.A-Sheild, darthmon_wmde, Premeditated, holger.knust, joker88john, CucyNoiD, Nandana, NebulousIris, Akovalyov, Tumz24, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, Jseddon, rosalieper, gabriel-wmde, DStrine, Wong128hk, JAllemandou, mobrovac, terrrydactyl, Wikidata-bugs, Snowolf, Base, aude, GWicke, Pcoombe, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, jeremyb, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T225195: EventBus jobs failing heavily because of CentralNotice and WikibaseRepo
Ottomata added a comment. So I suppose since those patches are in two different repos, tests on each fail until they are both merged? I think we should merge them. This is blocking other EventBus patches too. TASK DETAIL https://phabricator.wikimedia.org/T225195 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, AndyRussG, awight, thiemowmde, Aklapper, E.S.A-Sheild, darthmon_wmde, Premeditated, holger.knust, joker88john, CucyNoiD, Nandana, NebulousIris, Akovalyov, Tumz24, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, Jseddon, rosalieper, gabriel-wmde, DStrine, Wong128hk, JAllemandou, mobrovac, terrrydactyl, Wikidata-bugs, Snowolf, Base, aude, GWicke, Pcoombe, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, jeremyb, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T215413: Image Classification Working Group
Ottomata added a comment. Not sure if this is relevant, but this seemed the best place to note. I just came across: https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_YARN It seems relatively easy to package up (e.g. on a notebook host) and ship to hdfs and then include it in a spark job. TASK DETAIL https://phabricator.wikimedia.org/T215413 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Miriam, Ottomata Cc: Ottomata, Jheald, Cirdan, MoritzMuehlenhoff, CDanis, akosiaris, SandraF_WMF, Fuzheado, PDrouin-WMF, Krenair, d.astrikov, JoeWalsh, Nirzar, dcausse, fgiunchedi, JAllemandou, leila, Capt_Swing, mpopov, Nuria, DarTar, Halfak, Gilles, EBernhardson, dr0ptp4kt, Harej, MusikAnimal, Abit, elukey, diego, Cparle, Ramsey-WMF, Miriam, Isaac, darthmon_wmde, Premeditated, Nandana, JKSTNK, Akovalyov, Lahi, Gq86, E1presidente, Anooprao, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, V4switch, LawExplorer, Salgo60, Avner, Silverfish, _jensen, rosalieper, Susannaanas, Wong128hk, Jane023, terrrydactyl, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Matanya, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T212550: Implement support for ChronologyProtection in events sent when editing Mediawiki/Wikidata
Ottomata added a comment. I guess my comment here got missed? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/505328#message-f9c4c6e4cf0235e8dfa743ced1dc17ac6ca1dd21 TASK DETAIL https://phabricator.wikimedia.org/T212550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, Ottomata Cc: Krinkle, Ottomata, Aklapper, Smalyshev, aaron, alaa_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki
Ottomata added a comment. Can we close this task? TASK DETAIL https://phabricator.wikimedia.org/T161731 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, alaa_wmde, holger.knust, joker88john, ET4Eva, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Xinbenlv, Vacio, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T212550: Implement support for ChronologyProtection in RDF export
Ottomata added a comment. Hm, can we add some more documentation, especially to the field in the event schema? I'm not familiar with ChronologyProtector, and googling around for docs isn't yielding much. TASK DETAIL https://phabricator.wikimedia.org/T212550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper, Smalyshev, aaron, alaa_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T214706: Surface link changes as a stream
Ottomata added a comment. Just saw some errors in prod Failed processing event: Topic mediawiki.page-links-change not configured ^ should fix.TASK DETAILhttps://phabricator.wikimedia.org/T214706EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: EvanProdromou, Mvolz, Lydia_Pintscher, Addshore, gerritbot, Pchelolo, Nuria, mobrovac, Samwalton9, Ottomata, ssastry, Smalyshev, bearND, bmansurov, holger.knust, CucyNoiD, Nandana, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Avner, Lewizho99, Maathavan, _jensen, Agabi10, Eevans, JAllemandou, Hardikj, Wikidata-bugs, aude, GWicke, Capt_Swing, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T209655: Copy Wikidata dumps to HDFs
Ottomata added subscribers: Nuria, Ottomata.Ottomata added a comment. @Nuria we should fit this in somewhere! Maybe a Q3 goal? :DTASK DETAILhttps://phabricator.wikimedia.org/T209655EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Nuria, GoranSMilovanovic, Addshore, JAllemandou, bmansurov, Nandana, Akovalyov, Lahi, Gq86, QZanden, LawExplorer, Avner, _jensen, D3r1ck01, Wikidata-bugs, aude, Capt_Swing, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T210702: Verify that EventStreams work with WikiBase MediaInfo
Ottomata added a comment. Cool! EventStreams is deployed in beta, however, real production data does not go there. If your code that generates events is deployed to beta, you should be able to consume those event triggered by actions in beta from stream-beta.wmflabs.org. How will your events be generated? Are they just regular edits?TASK DETAILhttps://phabricator.wikimedia.org/T210702EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, MarkTraceur, Abbe98, Aklapper, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, E1presidente, Vacio, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, Tramullas, Acer, Susannaanas, Nirmos, JAllemandou, Jane023, Wikidata-bugs, Base, matthiasmullie, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, Steinsplitter, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T210451: Kafka eqiad.mediawiki.page-delete topic is empty
Ottomata added a comment. Oo, I just did the same, or, at least I copied the relevant files. They are on stat1004:/home/otto/eventbus-validation-logs0. Stas said he might have another way so I stopped there.TASK DETAILhttps://phabricator.wikimedia.org/T210451EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: MisterSynergy, mobrovac, TerraCodes, Liuxinyu970226, Aklapper, Pchelolo, Ottomata, Gehel, Smalyshev, Legado_Shulgin, Nandana, thifranc, Akovalyov, jijiki, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, D3r1ck01, Jonas, Xmlizer, Wong128hk, Eevans, JAllemandou, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, ArielGlenn, faidon, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, akosiaris, fgiunchedi, jeremyb, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T207817: WDQS Updater ran into issue and stopped working
Ottomata added a comment. Thanks @Smalyshev, I think you are write that changes like this should be announced a bit better. We'll try to do better next time. I wonder...where should we announce changes like this? engineering@? wikitech-l?TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Pchelolo, Tarrow, WMDE-leszek, Addshore, mmodell, Stashbot, 20after4, TerraCodes, Liuxinyu970226, gerritbot, Gehel, Smalyshev, Mathew.onipe, Aklapper, Dzahn, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, ArielGlenn, faidon, He7d3r, Jdforrester-WMF, Mbch331, Jay8g, fgiunchedi, greg___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T207817: WDQS Updater ran into issue and stopped working
Ottomata added a comment. Maybe jackson just can't parse this microsecond stuff? Maybe milliseconds are fine?TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Pchelolo, Tarrow, WMDE-leszek, Addshore, mmodell, Stashbot, 20after4, TerraCodes, Liuxinyu970226, gerritbot, Gehel, Smalyshev, Mathew.onipe, Aklapper, Dzahn, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T207817: WDQS Updater ran into issue and stopped working
Ottomata added subscribers: Pchelolo, Ottomata.Ottomata added a comment. Ping @Pchelolo. This is happening because of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/468482/ as part of a fix for T207329: Clear watchlist on enwiki only removes 50 items at a time. Interesting! I checked Jodatime stuff to make sure one of our Java based pipeline handled the timestamp format change, I'm surprised that Jackson can't parse this!TASK DETAILhttps://phabricator.wikimedia.org/T207817EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Pchelolo, Tarrow, WMDE-leszek, Addshore, mmodell, Stashbot, 20after4, TerraCodes, Liuxinyu970226, gerritbot, Gehel, Smalyshev, Mathew.onipe, Aklapper, Dzahn, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating
Ottomata added a comment. Ok, I've added the analytics-search system user to the analytics-search-users group. You should make your script chgrp analytics-search-users after it creates it.TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mpopov, OttomataCc: gerritbot, Gehel, Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating
Ottomata added a comment. Oh sorry, misunderstood. Yes we should be able to make the output of the file writable by you somehow.TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mpopov, OttomataCc: gerritbot, Gehel, Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating
Ottomata added a comment. @mpopov, since that file is managed by Puppet, you'll have to make a puppet patch to change it!TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: mpopov, OttomataCc: gerritbot, Gehel, Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, NebulousIris, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204415: Query stats dashboard not updating
Ottomata added a comment. all the Hive queries (and related) should be using 'webrequest_text' from now on. e.g. WHERE webrequest_source = 'text'TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Nuria, OttomataCc: Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T204415: Query stats dashboard not updating
Ottomata added a comment. See also T200822: Remove webrequest misc analytics related jobs and code after cache misc -> text merge is complete and T164609: Merge cache_misc into cache_text functionally. Sorry yall didn't know about this. I wonder if there is a better way we can configure datasources like this more dynamically. Hm.TASK DETAILhttps://phabricator.wikimedia.org/T204415EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Nuria, OttomataCc: Ottomata, elukey, Nuria, mpopov, chelsyx, Aklapper, Addshore, Smalyshev, Lydia_Pintscher, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Jonas, Xmlizer, JAllemandou, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki
Ottomata added a comment. OO yes @Smalyshev and in case you didn't see, we also increased retention of mediawiki topics to 31 days in the main kafka clusters.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Vacio, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days
Ottomata set the point value for this task to "2". TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a comment.
Doing the following for all main-eqiad and main-codfw:
for t in \
mediawiki.page-create\
mediawiki.page-delete\
mediawiki.page-edit \
mediawiki.page-move \
mediawiki.page-properties-change \
mediawiki.page-restrictions-change \
mediawiki.page-undelete \
mediawiki.page_delete\
mediawiki.page_move \
mediawiki.page_restore \
mediawiki.recentchange \
mediawiki.resource-change\
mediawiki.resource_change\
mediawiki.resourcechange \
mediawiki.revision-create\
mediawiki.revision-score \
mediawiki.revision-visibility-change \
mediawiki.revision_create\
mediawiki.revision_visibility_set\
mediawiki.user-blocks-change \
mediawiki.user_block ; do
for dc in eqiad codfw; do
topic="${dc}.${t}"
echo kafka configs --alter --entity-type topics --entity-name $topic --add-config retention.ms=267840
done
done
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-edit --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-edit --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-properties-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-properties-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-restrictions-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-restrictions-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page-undelete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page-undelete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_delete --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_move --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.page_restore --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.page_restore --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.recentchange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.recentchange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resource-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resource-change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resource_change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resource_change --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.resourcechange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.resourcechange --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name eqiad.mediawiki.revision-create --add-config retention.ms=267840
kafka configs --alter --entity-type topics --entity-name codfw.mediawiki.revision-create --add-config re
[Wikidata-bugs] [Maniphest] [Retitled] T187296: Increase kafka event retention to 31
Ottomata renamed this task from "Increase kafka event retention to 14 or 21 days" to "Increase kafka event retention to 31".Ottomata claimed this task. TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a comment. mediawiki eventbus topics should now be retained for 31 days in main Kafka clusters. If we add a new mediawiki topic, we need to remember to run this command for it.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a project: Analytics-Kanban. TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a comment. I'll make this 31 days just to bump it up to a month. We have plenty of space for this.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T187296: Increase kafka event retention to 14 or 21 days
Ottomata merged a task: T196409: Consider increasing retention for mediawiki event topics. TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a comment. Is there a reason we want to do this on main instead of jumbo? Stas will be consuming from jumbo, since it has timestamp offset support.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187296: Increase kafka event retention to 14 or 21 days
Ottomata added a comment. I think we can do this just for the mediawiki eventbus topics on the jumbo cluster.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86, Vacio, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Gehel, Jonas, FloNight, Xmlizer, Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T178445: flapping monitoring for recommendation_api on scb
Ottomata triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Joe, Volans, mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Qtn1293, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, LawExplorer, Avner, Zppix, FloNight, Eevans, Hardikj, Wikidata-bugs, aude, Capt_Swing, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T181988: Investigate and improve memory allocation rates of WDQS
Ottomata triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T181988EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Aklapper, Smalyshev, Gehel, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T177099: Large number of "A page you created was linked on Wikidata" emails to one recipient in short period of time
Ottomata triaged this task as "Low" priority. TASK DETAILhttps://phabricator.wikimedia.org/T177099EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Lydia_Pintscher, herron, Aklapper, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, grin, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T161731: Create reliable change stream for specific wiki
Ottomata added a comment. So, FYI, the timestamps as they are now are the timestamp that the kafka jumbo-eqiad cluster received the messages. These are replicated from the main-eqiad cluster, and might have a short (seconds usually, minutes max) delay. Eventually (work not planned yet) we will upgrade the other Kafka clusters, and also configure producers to set timestamps based content time, e.g. revision create timestamp, instead.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: gerritbot, JAllemandou, Pchelolo, Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Baloch007, Vacio, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Gehel, Jonas, FloNight, Xmlizer, Nirmos, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Manybubbles, Mbch331, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
