Zbyszko added a comment.
Thanks for bringing this up, we haven't explain that part very well yet. Goal of the new updater wasn't to have a best case scenario latency lower than on the old updater. Goal was to eliminate few issues of the old ones, like: - low throughput, that often caused huge spikes of lag that were very hard to get down from (this is a nice example - https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=8&orgId=1&from=1628010995348&to=1628065289312). - reliance on Blazegraph to reconcile the data - Blazegraphs read affect writes and vice versa, which quite often caused a cascading failure on both ends - update latency and query performance. - ineffective handling of eventual consistency - this was one of the reason for missing data in WDQS. New Updater was designed around those (and more) issues. Few things about it: - Higher best case lag is the effect of data reconciliation within the pipeline and it helps with the data completeness. We rather have a complete data set than have a faster incomplete one. - Data is reconciled within the pipeline, which has a dramatically lower effect on Blazegraph - this should help with the updates, which was the goal, but also positively affect query engine stability. - Max throughput of the current deployment is about 10x better than the old one - which means a much faster catch up and more room to grow for Wikidata. The new updater can be scaled even more - if we really need it to. We will provide more communication on this soon, along with some more background on the Streaming updater itself and the process of the development. TASK DETAIL https://phabricator.wikimedia.org/T294025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected]
