[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Gehel Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Maintenance_bot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 742770 **merged** by Ottomata: [operations/puppet@production] refine - bump jar version to get deduplication logic fix https://gerrit.wikimedia.org/r/742770 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, 786, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, 786, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 742770 had a related patch set uploaded (by Ottomata; author: Ottomata): [operations/puppet@production] refine - bump jar version to get deduplication logic fix https://gerrit.wikimedia.org/r/742770 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Maintenance_bot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735444 **merged** by Ottomata: [analytics/refinery/source@master] Refine - don't remove records during deduplication if ids are null https://gerrit.wikimedia.org/r/735444 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, 786, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735648 **merged** by Ottomata: [wikimedia-event-utilities@master] Always generate events with meta.id https://gerrit.wikimedia.org/r/735648 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735653 **merged** by jenkins-bot: [wikidata/query/rdf@master] Attach meta.id to side-output events https://gerrit.wikimedia.org/r/735653 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Gehel moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T294361 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Gehel Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735653 had a related patch set uploaded (by DCausse; author: DCausse): [wikidata/query/rdf@master] Attach meta.id to side-output events https://gerrit.wikimedia.org/r/735653 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735648 had a related patch set uploaded (by DCausse; author: DCausse): [wikimedia-event-utilities@master] Discourage generating events without meta.id https://gerrit.wikimedia.org/r/735648 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 4748kitoko, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
gerritbot added a comment. Change 735444 had a related patch set uploaded (by Ottomata; author: Ottomata): [analytics/refinery/source@master] Refine - don't remove records during deduplication if ids are null https://gerrit.wikimedia.org/r/735444 TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata added a comment. 3 our of your 7 records in this hour have meta.id == null (including this one), and Refine deduplication is removing all but one of them. You should set a unique value for `meta.id` in your events, but, arguably, Refine should not consider events with null `meta.id` as duplicates. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata added a comment. INTEResting. Will investigate. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Ottomata claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
odimitrijevic added a project: Data-Engineering-Kanban. odimitrijevic moved this task from Incoming to Ingest on the Data-Engineering board. TASK DETAIL https://phabricator.wikimedia.org/T294361 WORKBOARD https://phabricator.wikimedia.org/project/board/5462/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: odimitrijevic Cc: JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
odimitrijevic added a project: Data-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: odimitrijevic Cc: JAllemandou, dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: dcausse, Aklapper, EChetty, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 4748kitoko, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
dcausse added a parent task: T279541: Add a reconciliation strategy to the wdqs streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T294361 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: dcausse, Aklapper, EChetty, MPhamWMF, CBogen, 4748kitoko, Namenlos314, Akovalyov, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure
dcausse created this task.
dcausse added projects: Analytics, Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
While investigating missed updates on the WDQS streaming updater I looked at
our flink side-outputs that record all the failures happening while fetching
the data out of MW.
For instance I found that we missed the revision 1510811138 for the wikidata
entity Q6766777 (edit that occurred at `2021-10-11T17:11:00Z`).
The fetch for this entity had failed because of a 404 hitting MW (due to the
race between the events & mysql replication). This type of failure is recorded
in the topics [eqiad|codfw].rdf-streaming-updater.fetch-failure.
Looking at the raw data I can find this record:
hdfs dfs -text
/wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure/year=2021/month=10/day=11/hour=17/part.task_event_default_1633975518391_41_0.txt.gz
| grep Q6766777 | jq .
{
"meta": {
"domain": "www.wikidata.org",
"dt": "2021-10-11T17:11:00.853232Z",
"stream": "rdf-streaming-updater.fetch-failure"
},
"item": "Q6766777",
"original_ingestion_dt": "2021-10-11T17:11:00.732866Z",
"revision_id": 1510811138,
"original_event_info": {
"dt": "2021-10-11T17:11:00Z",
"$schema": "/mediawiki/revision/create/1.1.0",
"meta": {
"id": "b02f7b80-05d0-4060-9808-bcaef6a2304e",
"dt": "2021-10-11T17:11:00Z",
"stream": "mediawiki.revision-create",
"request_id": "d1297274-7e9e-4de9-a9f4-63d86d3b566a",
"domain": "www.wikidata.org"
}
},
"op_type": "diff",
"from_revision_id": 1392684585,
"exception_type":
"org.wikidata.query.rdf.tool.wikibase.WikibaseEntityFetchException",
"exception_msg": "Cannot fetch entity at
https://www.wikidata.org/wiki/Special:EntityData/Q6766777.ttl?flavor=dump&revision=1510811138:
ENTITY_NOT_FOUND",
"fetch_error_type": "ENTITY_NOT_FOUND",
"dt": "2021-10-11T17:11:00.853125Z",
"$schema": "/rdf_streaming_updater/fetch_failure/1.0.0"
}
But looking at the refined table I can't find this same record:
select item, meta.dt from event.rdf_streaming_updater_fetch_failure where
year = 2021 and month = 10 and day = 11 and item = "Q6766777";
finds nothing.
TASK DETAIL
https://phabricator.wikimedia.org/T294361
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dcausse
Cc: dcausse, Aklapper, EChetty, MPhamWMF, CBogen, 4748kitoko, Namenlos314,
Akovalyov, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer,
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984,
Manybubbles, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
