JanZerebecki added a subscriber: JanZerebecki. JanZerebecki added a comment.
The nature of these event type candidates is such that they are changes with a log existing at the provider. The only persistent state that each consuming service needs to know is the revision/time up to which it has applied the changes. The historic (recent) changes are already kept on the providing service, so a solution would not necessarily have to retain those. It is probably with a bit of care possible to apply changes out of order and in a way that it is idempotent so the consuming service could apply restartable and in parallel. The actual data can be pulled by the consumer from the provider; duplicating it in the queue probably has no benefit. I think the only thing here where pub/sub brings anything worth is the event that new changes are available to decrease latency of updates. Which means only ever one event in queue for all consumers, as new events contain all information of the previous ones. How is the job queue currently not reliable? ( https://www.mediawiki.org/wiki/Job_queue_redesign and https://www.mediawiki.org/wiki/Manual:Job_queue mention other problems but nothing regarding reliability problems. ) TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki Cc: GWicke, aaron, JanZerebecki, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe _______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
