gerritbot added a comment.
Change 448103 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use getClientLockName value for releaseClientLock when dispatching
https://gerrit.wikimedia.org/r/448103TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T20:39:50Z] Synchronized php-1.32.0-wmf.14/extensions/Wikibase/repo/includes/Store/Sql/SqlChangeDispatchCoordinator.php: Use getClientLockName value for releaseClientLock when dispatching T200420 (duration: 00m 57s)TASK
gerritbot added a comment.
Change 448113 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.14] Use getClientLockName value for releaseClientLock when dispatching
https://gerrit.wikimedia.org/r/448113TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Addshore added a comment.
The fix in the patch above has been on mwmaint1001 for around 15 mins now (I hacked it on there), and checked that the correct locks were being removed by name.
Dispatch seems to have recovered, now just need to back port it, and then we can investigate what actually
gerritbot added a comment.
Change 448113 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.14] Use getClientLockName value for releaseClientLock when dispatching
https://gerrit.wikimedia.org/r/448113TASK
gerritbot added a comment.
Change 448103 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Use getClientLockName value for releaseClientLock when dispatching
https://gerrit.wikimedia.org/r/448103TASK
Addshore added a comment.
@aaron has made a patch to make Lockmanager throw when trying to unlock a null lock @ https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/448080/
The issue in Wikibase should be fixed first though to avoid dispatching throwing exceptions and dieing & spamming logs.TASK
Addshore added a comment.
So, it looks like when lockManager->unlock is called in LockManagerSqlChangeDispatchCoordinator chd_lock is used, which is null, so no lock is removed in redis
I still have no idea how this triggered at 11am today, it looks like it will be due to .13 -> .14, but
aaron added a comment.
Ah, right, I read that ternary backwards, <<$maxTime < PHP_INT_MAX ? PHP_INT_MAX : 1>>.TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Nikki, zeljkofilipin, aaron, gerritbot,
Addshore added a comment.
In T200420#4453901, @aaron wrote:
In T200420#4453134, @Addshore wrote:
Something to note, because the locks are no longer in the DB, we end up selecting the same 15 or so wikis that are locked all of the time.
It could be that the other wikis actually don't have locks:
aaron added a comment.
In T200420#4453134, @Addshore wrote:
Something to note, because the locks are no longer in the DB, we end up selecting the same 15 or so wikis that are locked all of the time.
It could be that the other wikis actually don't have locks:
before using the redis lock manager
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T15:27:20Z] Synchronized wmf-config: Remove unused wikibaseDispatchRedisLockManager logging T200420 (duration: 00m 56s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T15:21:32Z] Synchronized wmf-config/InitialiseSettings.php: Add LockManager logging T200420 (duration: 00m 55s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Addshore added a comment.
I deployed T178652 which should hide some of the brokenness, as the locks should now only remain for 15 mins instead of an hour, but the underlying issue still remains.
I added a logger to the LockManager used for dispatching, but that hasn't logged anything yet...
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T14:38:52Z] Synchronized wmf-config/InitialiseSettings.php: Add wikibaseDispatchRedisLockManager to wmgMonologChannels T200420 (duration: 00m 54s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448043 merged by jenkins-bot:
[operations/mediawiki-config@master] Add wikibaseDispatchRedisLockManager to wmgMonologChannels
https://gerrit.wikimedia.org/r/448043TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T14:35:52Z] Synchronized wmf-config/Wikibase.php: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 56s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T14:34:37Z] sync-file aborted: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 02s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448042 merged by jenkins-bot:
[operations/mediawiki-config@master] Add a logger to wikibaseDispatchRedisLockManager
https://gerrit.wikimedia.org/r/448042TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448043 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Add wikibaseDispatchRedisLockManager to wmgMonologChannels
https://gerrit.wikimedia.org/r/448043TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448042 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Add a logger to wikibaseDispatchRedisLockManager
https://gerrit.wikimedia.org/r/448042TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T13:44:00Z] Synchronized wmf-config/InitialiseSettings.php: Use new wikibase dispatch lock manager on wikidatawiki T200420 T178652 (duration: 00m 55s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448037 merged by Alexandros Kosiaris:
[operations/puppet@production] Verbose logging for wikidata dispatching
https://gerrit.wikimedia.org/r/448037TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448037 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/puppet@production] Verbose logging for wikidata dispatching
https://gerrit.wikimedia.org/r/448037TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T12:44:01Z] Synchronized wmf-config/Wikibase.php: T200420 - Wikidata, dispatch, select 20 instead of 15 wikis (duration: 00m 55s)TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448034 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikidata dispatch, select 20 wikis instead of 15
https://gerrit.wikimedia.org/r/448034TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
gerritbot added a comment.
Change 448034 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Wikidata dispatch, select 20 wikis instead of 15
https://gerrit.wikimedia.org/r/448034TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-07-26T12:38:03Z] rebuilt and synchronized wikiversions files: wikidatawiki back to .13 T200420TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL
Addshore added a comment.
Something to note, because the locks are no longer in the DB, we end up selecting the same 15 or so wikis that are locked all of the time.
It could be that the other wikis actually don't have locks:
12:34:54 Could not lock any of the candidate client wikis for
29 matches
Mail list logo
