It seems as of November 29th, something in our logging infrastructure has
stopped.
Does anyone know what happened?
Debugging:
Looking at the graphs[1] reveals all data points have them plummeted straight
down that day and haven't shown any activity since.
Except on December 4th we observed that the unrealistic value of "0" stopped
begin echoed as well. After that, the line is absent entirely.
Example:
http://graphite.wikimedia.org/render/?title=addOnloadHook&width=900&height=200&target=aliasByNode(smartSummarize(mw.js.deprecate.addOnloadHook.rate%2C%221d%22%2C%22avg%22)%2C-1)&from=-4weeks
Raw count:
http://graphite.wikimedia.org/render/?width=900&height=300&target=mw.js.deprecate.addOnloadHook.count&from=-4weeks
The software stack:
* graphite:statsd:
-
https://github.com/wikimedia/operations-puppet/commits/3a1134921/modules/txstatsd
-
https://github.com/wikimedia/operations-puppet/commits/3a1134921/modules/webperf
* EventLogging: https://meta.wikimedia.org/wiki/Schema:DeprecatedUsage
* WikimediaEvents:
https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/34e42e1d73/modules/ext.wikimediaEvents.deprecate.js
* mw.log.deprecate:
https://github.com/wikimedia/mediawiki/blob/c6131c5df4/resources/src/mediawiki/mediawiki.js#L564-L573
Nothing there stood out to me.
When connecting to tcp://vanadium.eqiad.wmnet:8600 via zmq from tin.eqiad.wmnet
directly[2], there's still a steady flow of incoming data via EventLogging. So
it must've stagnated somewhere further down the line (on servers I don't have
access to).
Best,
— Timo
[1] mw.js.deprecate graphs:
last 5 months: http://codepen.io/Krinkle/full/zyodJ/
last 3 weeks: http://codepen.io/Krinkle/full/cBGCl/
[2] https://gist.github.com/Krinkle/764cbbadbe272c7ca1e0
_______________________________________________
Analytics mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/analytics
