[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

gerritbot added a comment. Change 249222 abandoned by Gergő Tisza: Make nutcracker's auto_eject_hosts setting configurable Reason: not relevant anymore. https://gerrit.wikimedia.org/r/249222 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

gerritbot added a comment. Change 289582 merged by jenkins-bot: Retry after OAuth "Nonce already used" error https://gerrit.wikimedia.org/r/289582 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

gerritbot added a comment. Change 289582 had a related patch set uploaded (by John Vandenberg): Retry after OAuth "Nonce already used" error https://gerrit.wikimedia.org/r/289582 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

Tgr added a comment. @jayvdb: I don't have much time this month but do you want to shepherd https://gerrit.wikimedia.org/r/#/c/249222/ through a puppet SWAT (it needs a rebase too) and then disable the setting in beta? That's my best bet at improving the situation. The problem is somewhere

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

gerritbot added a comment. Change 249222 had a related patch set uploaded (by Gergő Tisza): Make nutcracker's auto_eject_hosts setting configurable https://gerrit.wikimedia.org/r/249222 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

XZise added a comment. Primarily a response from @jayvdb to the comments in https://gerrit.wikimedia.org/r/#/c/237977/. And as soon as that got merged we are going to get more information from Travis. Not sure who can then help with that information though. TASK DETAIL

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

Tgr added a comment. Per graphite , there are tens of thousands of memcached failures per day. Maybe @ori has ideas, he debugged a

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

Aklapper added a comment. This task has "Unbreak now" priority for four weeks now. How to get this moving forward? Which info is needed and from who? TASK DETAIL

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

XZise added a subscriber: XZise. XZise added a comment. Not sure if that patch helped but since a few days all tests on Travis connecting to OAuth fail similar to https://travis-ci.org/wikimedia/pywikibot-core/jobs/81789795#L1016 And I can't find servedby anywhere there (except for the 5 times

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

gerritbot added a subscriber: gerritbot. gerritbot added a comment. Change 237977 had a related patch set uploaded (by John Vandenberg): Expand login and rights exception tree https://gerrit.wikimedia.org/r/237977 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

jayvdb added a comment. @Tgr, I am not sure what we can or should do to help, but if you can give us some suggestions we might be able to help. It is occurring all over the place, at any test, before or after the first login. To give you some idea of frequency, each test job performs about 40

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

Tgr added a comment. Looks like the memcached graphite counts are a good way of tracking this after all (nutcracker restart was around 18:30 UTC): F2563030: beta memcached errors.png TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: "Nonce already used" regularly occurring on beta cluster

Tgr added a comment. @jayvdb is there a way to track this error? I looked at the travis history, but builds seem to break all over the place and mostly for unrelated reasons (or not obviously related, anyway - pretty much any error can be related to memcached failure), so I couldn't get a good

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. https://travis-ci.org/jayvdb/pywikibot-core/jobs/78718812#L721 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb Cc: Tgr, greg, Aklapper, Luke081515,

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. Another one on beta cluster: https://travis-ci.org/jayvdb/pywikibot-core/jobs/77788314#L1004 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb Cc: Tgr, greg,

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. Another, beta cluster: https://travis-ci.org/wikimedia/pywikibot-core/jobs/77787307#L741 This happens so frequently, and in such varied code paths, that I would say Oauth on the beta cluster is effectively broken for any non-trivial amount of automated testing. TASK

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. A bunch of failures at https://travis-ci.org/jayvdb/pywikibot-core/jobs/77671728#L726 and https://travis-ci.org/jayvdb/pywikibot-core/jobs/77671725#L1228 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

greg added a subscriber: greg. greg added a comment. From my understanding, this is an issue in OAuth not Beta Cluster, yes? I'm removing the Beta Cluster project as such. TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. We have are testing oauth on multiple sites (see task description), and we have only seen this on the beta cluster. TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Pywikipedia-bugs] [Maniphest] [Commented On] T109173: Nonce already used regularly occurring on beta cluster

jayvdb added a comment. Another one: https://travis-ci.org/jayvdb/pywikibot-core/jobs/77589690#L1002 TASK DETAIL https://phabricator.wikimedia.org/T109173 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jayvdb Cc: Aklapper, Luke081515,