GWicke added a comment. In https://phabricator.wikimedia.org/T84923#940155, @JanZerebecki wrote:
> The nature of these event type candidates is such that they are changes with > a log existing at the provider. Wikidata might be the exeception here. Most other events are not available at the provider in a reliable manner. > The actual data can be pulled by the consumer from the provider; Is there an API for this already? > I think the only thing here where pub/sub brings anything worth is the event > that new changes are available to decrease latency of updates. Which means > only ever one event in queue for all consumers, as new events contain all > information of the previous ones. A major benefit of an event stream is reliability, performance and simplicity. The same generic client (using websockets for example) can be used to consume a variety of events. Queues like Kafka are optimized for the use case and perform really well at high request volumes. As an example, we are currently processing about 50k messages/second (request logs) with two Kafka nodes. I know that most other events are lower volume than this, but it's good to have headroom in the system. > How is the job queue currently not reliable? For one, it uses redis for storage, which only supports async master/slave replication and has limited options for durability. Async replication means that you are likely to lose messages in a fail-over. Limited durability means that you lose messages on power loss. It also does not support multiple consumers for each event (no pub/sub), which results in fairly static coupling of producer and consumer. You need to create a new job per consumer. Finally, the job runners are doing all kinds of processing instead of pure event delivery. As mis-behaving jobs compete with others for resources, this makes them less reliable than a pure event delivery system would be. TASK DETAIL https://phabricator.wikimedia.org/T84923 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: GWicke, aaron, JanZerebecki, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, RobH, aude, Manybubbles, daniel, mark, Joe _______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
