fgiunchedi added a comment.
In T221774#5155621 <https://phabricator.wikimedia.org/T221774#5155621>, @hoo wrote: > Possible way to do this: > > Create `PrometheusBlazegraphLagService` class which internally fetches the lag from a given Blazegraph instance like `curl "http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=scalar(time()%20-%20blazegraph_lastupdated%7Binstance%3D%22wdqs1005.eqiad.wmnet%3A9193%22%7D)"` (where `wdqs1005.eqiad.wmnet` is to be replaced by the hostname). That would be cached (given we don't want to hit Prometheus often and as we care for lag in the 30-60m range, fetching this once ever 1-5m should be fine… this could maybe even be done in a Job). We would do that for all known `wdqs` instances and then sum/average/… the results. > > This value would then be used for adjusting maxlag, as described above. > > Things to consider: > > - Is going to Prometheus directly the right thing to do? > - How often can we sanely hit Prometheus? > - Where do we want to manage the list of WDQS instance for this? (Or I guess can we also ask Prometheus for all metrics at once?) re: frequency even once a minute would be fine, since the query isn't heavy to run, and yes you can ask about all instances at once, or e.g. take the `max()`. Which leads me to a question re: servers in maintenance, where / how is the list maintained or will be maintained of all instances and/or instances in maintenance? I'm asking because if the list of instances that should be queried is known anyways IMHO it'd be simpler to query the lag via sparql and keep the prometheus out of the loop entirely. I'm saying this because IIRC the "lastupdated" value would go blazegraph -> prometheus -> mediawiki TASK DETAIL https://phabricator.wikimedia.org/T221774 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, fgiunchedi Cc: Ladsgroup, Smalyshev, fgiunchedi, hoo, Daniel_Mietchen, MisterSynergy, Addshore, Sjoerddebruin, Aklapper, Lucas_Werkmeister_WMDE, darthmon_wmde, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
