[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-08 Thread jijiki
jijiki added a comment. Looking at mc1035's behaviour after reimaging, it appears that its hit ratio reached 0.95 after ~10h, and it reached 28Mil items in ~13h. I think it is safe to say that we can reimage one memcached server in each DC every 12 hours. F33939723: image.png

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-08 Thread gerritbot
gerritbot added a comment. Change 646967 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli): [operations/puppet@production] hiera: remove shard16 from redis, reimage mc1034/mc2034 to buster https://gerrit.wikimedia.org/r/646967 TASK DETAIL

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-08 Thread jijiki
jijiki updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-08 Thread jijiki
jijiki updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['mc1035.eqiad.wmnet'] and were **ALL** successful. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: mc1035.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202012071923_jiji_17919_mc1035_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['mc2035.codfw.wmnet'] and were **ALL** successful. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: mc2035.codfw.wmnet The log can be found in `/var/log/wmf-auto-reimage/202012071834_jiji_1742_mc2035_codfw_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['mc2035.codfw.wmnet'] Of which those **FAILED**: ['mc2035.codfw.wmnet'] TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread ops-monitoring-bot
ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: mc2035.codfw.wmnet The log can be found in `/var/log/wmf-auto-reimage/202012071810_jiji_12811_mc2035_codfw_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread gerritbot
gerritbot added a comment. Change 646751 **merged** by Effie Mouzeli: [operations/puppet@production] hiera: remove shard17 from redis, reimage mc1035/mc2035 to buster https://gerrit.wikimedia.org/r/646751 TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread gerritbot
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread gerritbot
gerritbot added a comment. Change 646751 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli): [operations/puppet@production] hiera: remove shard17 from redis, reimage mc1035/mc2035 to buster https://gerrit.wikimedia.org/r/646751 TASK DETAIL

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-12-07 Thread jijiki
jijiki added a comment. Given that the total size of our redis cluster is now ~2G ( as discussed in T252391#6647730 ) we can upgrade a few more memcached servers to buster while we ensure that our work towards using redis 2.8 to buster is

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-11-06 Thread hashar
hashar closed subtask T267388: Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hashar Cc: Ladsgroup, Krenair,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-11-06 Thread hashar
hashar added a subtask: T267388: Puppet failure on deployment-memc08.deployment-prep.eqiad.wmflabs . TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hashar Cc: Ladsgroup, Krenair, elukey, jijiki,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread jijiki
jijiki added a comment. @aaron thank you! I updated the task description TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread jijiki
jijiki updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread jijiki
jijiki added a comment. @aaron If you have any insights regarding the Redis Lock Manager and file upload, it would be much appreciated (+ T265643 ) TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread aaron
aaron added a comment. Regarding RedisLockManager (it only needs 2 of the 3 host to be reachable). If one of them is depooled or refuses connections, no one should notice any disruption. For otherwise unreachable servers, there is a 2 second timeout (and the redis server will be avoided for

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread jijiki
jijiki updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe,

[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster

2020-10-28 Thread jijiki
jijiki added a subscriber: Ladsgroup. jijiki added a project: Wikidata. jijiki updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: Ladsgroup, Krenair, elukey,