| Ottomata added a comment. |
If so, you may want to consider consuming from Kafka rather than EventStreams.
I am considering this too, but I assume it's more code for me to write (maybe wrongly, I didn't look at it closely).
It will be more, a lot more. What language are you working in?
But what in fact I need is something like 10-20 parallel consumers
Load balanced parallel consumers, or all distinct consumers doing different stuff?
Nice thing about using a Kafka client directly, is you can subscribe to multiple topics/partitions in a single consumer group, and have the load automatically balanced between them (up to the total number of topic-partitions). If any of those processes goes down, a different consumer process will be auto-assigned its work.
TASK DETAIL
EMAIL PREFERENCES
To: Ottomata
Cc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb
Cc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
