| Smalyshev added a comment. |
let's jump in a hangout sometime to discuss this more.
Would be glad to. I'll try to set up something next week.
If there is a desire, we can expose these in EventStreams. Do you have desire? :)
Yes, see T145712 - recentchanges ignores pageprops updates, and it would be nice to have those too.
Does WDQS run in production?
Yes.
If so, you may want to consider consuming from Kafka rather than EventStreams.
I am considering this too, but I assume it's more code for me to write (maybe wrongly, I didn't look at it closely).
Kafka consumers support parallelization and scale much better than the EventStreams HTTP Kafka consumer proxy.
I may have written requirements to ask for more than I need, it gives an impression it has to serve Wikipedia-scale consumer load maybe. But what in fact I need is something like 10-20 parallel consumers, that order of magnitude. So I don't really foresee too much problem here as long as it doesn't require doing unindexed DB queries. I expect both Kafka and ES be fine with it.
Cc: Anomie, Aklapper, Smalyshev, QZanden, EBjune, merbst, Salgo60, Avner, debt, Gehel, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, JAllemandou, mobrovac, jkroll, Wikidata-bugs, Jdouglas, aude, GWicke, Deskana, Manybubbles, Mbch331, Krenair, jeremyb
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
