[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-15 Thread gerritbot
gerritbot added a comment. Change 467331 merged by Gehel: [operations/puppet@production] wdqs: re-enable kafka poller on wdqs public cluster https://gerrit.wikimedia.org/r/467331TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-15 Thread gerritbot
gerritbot added a comment. Change 467331 had a related patch set uploaded (by Gehel; owner: Gehel): [operations/puppet@production] wdqs: re-enable kafka poller on wdqs public cluster https://gerrit.wikimedia.org/r/467331TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-11 Thread gerritbot
gerritbot added a comment. Change 466722 merged by Gehel: [operations/puppet@production] wdqs: use recent change poller on public cluster instead of kafka https://gerrit.wikimedia.org/r/466722TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-11 Thread gerritbot
gerritbot added a comment. Change 466722 had a related patch set uploaded (by Gehel; owner: Gehel): [operations/puppet@production] wdqs: use recent change poller on public cluster instead of kafka https://gerrit.wikimedia.org/r/466722TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-09 Thread Smalyshev
Smalyshev added a comment. So I tried to run it with RC updater, and it seems to be catching up much faster than with Kafka updater. Which can mean: a. Kafka has way more events that RC updater ignores b. Kafka is not reporting offsets accurately c. There's some bug that makes Kafka poller slower

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-09 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-09T10:49:54Z] repooling wdqs2001 catched up on lag - T206423TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot,

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-09 Thread Gehel
Gehel added a comment. Looking at dropped packets, it looks like we did not have any over the last few days. So we have another cause to our lag. Also not that while the issue still seems more present on wdqs2003, we also see issue with other nodes.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-09 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-09T08:06:19Z] depooling wdqs2001 to catch up on lag -T206423TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot,

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-08 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-08T20:42:26Z] repooling wdqs2003 catched up on lag - T206423TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot,

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-08 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-08T19:08:48Z] depooling wdqs2003 to catch up on lag -T206423TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot,

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-08 Thread Gehel
Gehel added a comment. Looking at Grafana I can see spikes in batch progress that correlate with drops in lag. Zooming in, I can even see negative drops into batch progress, which should not happen. I suspect our metrics are skewed by the non monotonic nature of kafka updates (just a guess). Since

[Wikidata-bugs] [Maniphest] [Commented On] T206423: The usual Lag pattern for wdqs2003 seems to be taking another turn

2018-10-08 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-10-08T07:38:20Z] reducing relative weight of wdqs2003 in pybal - T206423TASK DETAILhttps://phabricator.wikimedia.org/T206423EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: