[Wikidata-bugs] [Maniphest] T294025: WDQS lag is usually longer than one minute
Zbyszko closed this task as "Resolved". Zbyszko claimed this task. Zbyszko added a comment. More information provided on wikidata-tech,wikidata mailing lists - https://lists.wikimedia.org/hyperkitty/list/wikidata-t...@lists.wikimedia.org/thread/O5UWOENIEDA2GJSDBL3MEBBFH7ZPXFNY/ TASK DETAIL https://phabricator.wikimedia.org/T294025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Addshore, Zbyszko, Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294025: WDQS lag is usually longer than one minute
Zbyszko added a comment. Thanks for bringing this up, we haven't explain that part very well yet. Goal of the new updater wasn't to have a best case scenario latency lower than on the old updater. Goal was to eliminate few issues of the old ones, like: - low throughput, that often caused huge spikes of lag that were very hard to get down from (this is a nice example - https://grafana.wikimedia.org/d/00489/wikidata-query-service?viewPanel=8=1=1628010995348=1628065289312). - reliance on Blazegraph to reconcile the data - Blazegraphs read affect writes and vice versa, which quite often caused a cascading failure on both ends - update latency and query performance. - ineffective handling of eventual consistency - this was one of the reason for missing data in WDQS. New Updater was designed around those (and more) issues. Few things about it: - Higher best case lag is the effect of data reconciliation within the pipeline and it helps with the data completeness. We rather have a complete data set than have a faster incomplete one. - Data is reconciled within the pipeline, which has a dramatically lower effect on Blazegraph - this should help with the updates, which was the goal, but also positively affect query engine stability. - Max throughput of the current deployment is about 10x better than the old one - which means a much faster catch up and more room to grow for Wikidata. The new updater can be scaled even more - if we really need it to. We will provide more communication on this soon, along with some more background on the Streaming updater itself and the process of the development. TASK DETAIL https://phabricator.wikimedia.org/T294025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Zbyszko, Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294025: WDQS lag is usually longer than one minute
Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T294025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, Bugreporter, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294025: WDQS lag is usually longer than one minute
Bugreporter created this task. Bugreporter added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION The old updater: https://grafana.wikimedia.org/d/00489/wikidata-query-service?viewPanel=8=1=162950400=162950760 (usually the lag is less than one minute) The new updater: https://grafana.wikimedia.org/d/00489/wikidata-query-service?viewPanel=8=1=163477440=163477800 (usually the lag is more than one minute) This indicates the new updater is //not// an improvement. TASK DETAIL https://phabricator.wikimedia.org/T294025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bugreporter Cc: Aklapper, Bugreporter, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
