[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
MZMcBride added a subscriber: MZMcBride. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MZMcBride Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, daniel, Krenair, bd808, MZMcBride, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, marcoil, Manybubbles, mark, RobLa-WMF, Joe, QChris, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
bd808 added a subscriber: bd808. bd808 added a comment. can support large delays (order of days) for individual consumers Do you have a strong use case to support this need? Kafka may very well be able to support this but I'm wondering it there is a specific and strong reason for such a long duration for an event bus. This seems like a specification for an event based storage system rather than a comm bus. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bd808 Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, daniel, Krenair, bd808, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, mark, RobLa-WMF, Joe, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
Krenair added a subscriber: Krenair. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Krenair Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, daniel, Krenair, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, mark, Joe, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
JanZerebecki added a subscriber: daniel. JanZerebecki added a comment. While talking to @daniel he noticed that I missed that the actual serialized diff is saved in https://www.mediawiki.org/wiki/Wikibase/Schema/wb_changes there is just no API. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, daniel, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, mark, Joe, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
Hardikj added a subscriber: Hardikj. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Hardikj Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, Hardikj, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
yuvipanda added subscribers: Halfak, yuvipanda. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: yuvipanda Cc: GWicke, aaron, JanZerebecki, mobrovac, Halfak, yuvipanda, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe, chasemp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
mobrovac added a subscriber: mobrovac. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: mobrovac Cc: GWicke, aaron, JanZerebecki, mobrovac, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed CC] T84923: Reliable publish / subscribe event bus
JanZerebecki added a subscriber: JanZerebecki. JanZerebecki added a comment. The nature of these event type candidates is such that they are changes with a log existing at the provider. The only persistent state that each consuming service needs to know is the revision/time up to which it has applied the changes. The historic (recent) changes are already kept on the providing service, so a solution would not necessarily have to retain those. It is probably with a bit of care possible to apply changes out of order and in a way that it is idempotent so the consuming service could apply restartable and in parallel. The actual data can be pulled by the consumer from the provider; duplicating it in the queue probably has no benefit. I think the only thing here where pub/sub brings anything worth is the event that new changes are available to decrease latency of updates. Which means only ever one event in queue for all consumers, as new events contain all information of the previous ones. How is the job queue currently not reliable? ( https://www.mediawiki.org/wiki/Job_queue_redesign and https://www.mediawiki.org/wiki/Manual:Job_queue mention other problems but nothing regarding reliability problems. ) TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki Cc: GWicke, aaron, JanZerebecki, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
