Smalyshev added a comment.
One thing to consider here to stop the situation getting too terrible would be to add the wdqs lag to the maxlag for wikidata.org
@Addshore I am not very informed on this one, what's maxlag and how it works?TASK DETAILhttps://phabricator.wikimedia.org/T209201EMAIL
Smalyshev added a comment.
So it sounds like the only place to fix this is within blazegraph itself.?
One of the solutions may be to try and figure out how to do faster updates. Another would be to add servers to production cluster to spread query load. The pattern seems to be very dependant on
Smalyshev added a comment.
Could it be related to this report
We are not using RC changes now, but Kafka stream, so not likely.
I wonder if running them on a different host might provide a small bit of processing relief for the wdqs hosts?
Updater process does not consume a lot of CPU (most
Tarrow added a comment.
Could it be related to this report: https://lists.wikimedia.org/pipermail/wikidata/2018-November/012610.html ?TASK DETAILhttps://phabricator.wikimedia.org/T209201EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: TarrowCc: Tarrow,
Addshore added a comment.
Regarding what I said in T209201#4738324 re adding the wdqs lag to maxlag... that could be slightly tricky.
Right now the options for getting the maxlag are querying each server individually, or querying prometheus.
Also, waiting on a external service to return a result
Addshore added a comment.
Looking at the edit rate, and page creation rate during the time period that the query service seemed to struggle updating it doesn't look like there was any real increase for the ~20 hours in question (10 hours of which the lag was going up and 10 hours of which the lag
Addshore added a comment.
One thing to consider here to stop the situation getting too terrible would be to add the wdqs lag to the maxlag for wikidata.org
The edit rate on wikidata drops massively when the maxlag hits 5 for either DBs or the dispatching process.
F27192963: image.pngTASK
Smalyshev added a comment.
Looking at the servers, we have very low update throughput:
01:17:43.485 [main] INFO org.wikidata.query.rdf.tool.Updater - Polled up to 2018-11-09T23:13:34Z at (3.5, 3.9, 1.9) updates per second and (543.7, 598.2, 291.9) milliseconds per second
Which probably means
