[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata closed subtask T273901: Automate event stream ingestion into HDFS for 
streams that dont use EventGate as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  @Gehel @dcausse these events are now in HDFS.  There aren't any Hive tables 
yet because no non-canary events have yet been ingested, and we filter out 
canary events from the Hive tables.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 668119 **merged** by jenkins-bot:
  [operations/mediawiki-config@master] Set canary_events_enabled: true for 
rdf-streaming-updater streams
  
  https://gerrit.wikimedia.org/r/668119

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 668119 had a related patch set uploaded (by Ottomata; owner: Ottomata):
  [operations/mediawiki-config@master] Set canary_events_enabled: true for 
rdf-streaming-updater streams
  
  https://gerrit.wikimedia.org/r/668119

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
abian, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Gehel]

Gehel closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Gehel
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Maintenance_bot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 
Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, 
Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-02-10T15:26:25Z] 
 Synchronized wmf-config/InitialiseSettings.php: Do not 
produce canary events for rdf-streaming-updater streams - T269619 
 (duration: 01m 13s)

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Stashbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 663219 **merged** by jenkins-bot:
  [operations/mediawiki-config@master] Do not produce canary events for 
rdf-streaming-updater streams
  
  https://gerrit.wikimedia.org/r/663219

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 663219 had a related patch set uploaded (by Ottomata; owner: Ottomata):
  [operations/mediawiki-config@master] Do not produce canary events for 
rdf-streaming-updater streams
  
  https://gerrit.wikimedia.org/r/663219

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Maintenance_bot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 
Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, 
Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-02-10T12:26:44Z] 
 Synchronized wmf-config/InitialiseSettings.php: T269619 
: [wdqs] Add flink sideoutput stream 
definitions (duration: 01m 06s)

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Stashbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 661727 **merged** by jenkins-bot:
  [operations/mediawiki-config@master] [wdqs] Add flink sideoutput stream 
definitions
  
  https://gerrit.wikimedia.org/r/661727

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 649715 **merged** by jenkins-bot:
  [wikidata/query/rdf@master] Add json encoders for side output events
  
  https://gerrit.wikimedia.org/r/649715

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-analytics) [2021-02-08T15:11:53Z]  set 
kafka topic retention to 31 days for 
(eqiad|codfw.rdf-streaming-updater.mutation) in kafka main-eqiad and main-codfw 
- T269619 

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Stashbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-02-08T15:11:47Z]  
set kafka topic retention to 31 days for 
(eqiad|codfw.rdf-streaming-updater.mutation) in kafka main-eqiad and main-codfw 
- T269619 

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Stashbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 661727 had a related patch set uploaded (by DCausse; owner: DCausse):
  [operations/mediawiki-config@master] [wdqs] Add flink sideoutput stream 
definitions
  
  https://gerrit.wikimedia.org/r/661727

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 647723 **merged** by Ottomata:
  [schemas/event/secondary@master] Add rdf-streaming-updater schemas for side 
outputs
  
  https://gerrit.wikimedia.org/r/647723

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  T270371  is great thank you!
  
  > If this kind of usecases is something we'd like to support
  
  We do. EventGate is a poor substitute for a full fledged Kafka client.  Kafka 
has so many more features and knobs to dial, and for prod apps those are going 
to be important.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[dcausse]

dcausse added a comment.


  In T269619#6696071 , 
@Ottomata wrote:
  
  > It depends on what you want to do :) EventGate will handle multi DC, 
filling some default values, and topic prefixes for you, but is an extra hop to 
Kafka.  As a prod system in a language with a good Kafka client, producing to 
Kafka is totally allowed.  You'd be the first main user of event platform not 
going through EventGate, but it is definitely something we want to support.
  >
  > Perhaps we'll want to build in some logic for doing some of the things 
EventGate is doing (as a proxy) into wikimedia-event-utilties (including 
validation, as you suggested).
  
  I think the most important for me is that these events get stored in HDFS 
properly and I'm not sure what are the requirements for this. I went with 
event-gate because I know it works :)
  
  But using the flink kafka producers makes more sense, bit's robust and easy 
to use but I was worried of missing important parts of the process. If this 
kind of usecases is something we'd like to support then I think I should go 
with this approach, I created T270371 
 to discuss all the tools that 
wikimedia-event-utilties should provide to help this usecase.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  It depends on what you want to do :) EventGate will handle multi DC, filling 
some default values, and topic prefixes for you, but is an extra hop to Kafka.  
As a prod system in a language with a good Kafka client, producing to Kafka is 
totally allowed.  You'd be the first main user of event platform not going 
through EventGate, but it is definitely something we want to support.
  
  Perhaps we'll want to build in some logic for doing some of the things 
EventGate is doing (as a proxy) into wikimedia-event-utilties (including 
validation, as you suggested).

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[dcausse]

dcausse added a comment.


  In T269619#6695454 , 
@Ottomata wrote:
  
  > @dcausse, will these be POSTed to an EventGate, or to produced directly to 
Kafka?
  
  I plan to POST them to event gate using a very naive SinkFunction and see how 
it behaves but I can push directly to kafka if it's preferable?

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  @dcausse, will these be POSTed to an EventGate, or to produced directly to 
Kafka?

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  > would it make sense to add some helper functions to 
wikimedia-event-utilities for validating a json against its schema
  
  Sure that could be useful :)

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[dcausse]

dcausse added a comment.


  In T269619#6693136 , 
@Ottomata wrote:
  
  > @dcausse for retrieving schemas, 
https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia-event-utilities/+/refs/heads/master
 might help. :)
  
  Thanks!
  it worked like a charm but I still had to pull 
`com.github.java-json-tools:json-schema-validator:2.2.14` to do schema 
validation, would it make sense to add some helper functions to 
`wikimedia-event-utilities` for validating a json against its schema? Use-case 
is a unit test to make sure that the json produced is compliant with the schema 
it's referencing.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Ottomata]

Ottomata added a comment.


  @dcausse for retrieving schemas, 
https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia-event-utilities/+/refs/heads/master
 might help. :)

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Ottomata
Cc: Ottomata, Aklapper, Gehel, Mstyles, MPhamWMF, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 649715 had a related patch set uploaded (by DCausse; owner: DCausse):
  [wikidata/query/rdf@master] [WIP] Add json encoders for side output events
  
  https://gerrit.wikimedia.org/r/649715

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[gerritbot]

gerritbot added a comment.


  Change 647723 had a related patch set uploaded (by DCausse; owner: DCausse):
  [schemas/event/secondary@master] Add rdf-streaming-updater schemas for side 
outputs
  
  https://gerrit.wikimedia.org/r/647723

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, gerritbot
Cc: Ottomata, Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Mstyles]

Mstyles added a subscriber: Ottomata.
Mstyles added a comment.


  @Ottomata also suggested via IRC to consider using the event platform instead 
of kafka

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Mstyles
Cc: Ottomata, Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T269619: Create pipelines for late/spurious/failed events

[Mstyles]

Mstyles created this task.
Mstyles added projects: Wikidata, Wikidata-Query-Service, Discovery-Search 
(Current work).

TASK DESCRIPTION
  As a production service, the flink streaming updater job needs somewhere to 
send events that are late/failed/spurious. Currently these go to HDFS but apps 
in the Kubernetes cluster can't talk to the Analytics cluster. There needs to 
be a Kafka bridge that the rdf-streaming-updater can send events to, and those 
events will eventually end up in HDFS.
  
  Acceptance Criteria:
  Kafka topics are set up for late/failed/spurious events
  Those same events end up in HDFS

TASK DETAIL
  https://phabricator.wikimedia.org/T269619

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, Mstyles
Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs