[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. In T188045#4045150, @Platonides wrote: Well, if the server itself is needed, it will be doing its work with a different IP address than the one of wdqs1004, since it would have been suffering the same connection issues as wdqs1004 (just the other half time). The other server was decommissioned, not needed! So no worries.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Platonides, MoritzMuehlenhoff, TerraCodes, faidon, Liuxinyu970226, ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, ayounsi, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, mark, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Platonides added a comment. Well, if the server itself is needed, it will be doing its work with a different IP address than the one of wdqs1004, since it would have been suffering the same connection issues as wdqs1004 (just the other half time).TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, PlatonidesCc: Platonides, MoritzMuehlenhoff, TerraCodes, faidon, Liuxinyu970226, ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, ayounsi, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, mark, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. Ok, I misparsed all of that. So next steps: Chris traces out and sees what is connected to ge-4/0/18. It somehow has the same IP address as wdqs1004 and is just sitting there. If we can, we shut down that server. If we cannot, we fix dns so its pointing correctly. If wdqs1004 needs a new ip, update dns and reimage. TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: MoritzMuehlenhoff, TerraCodes, faidon, Liuxinyu970226, ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, ayounsi, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, mark, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment. @Cmjohnson did message me about changing the port as well, so that's probably not it. Side note if it wasnt clear from the gerrit change above, OS also has been upgraded from jessie to stretch as part of the reimage.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, GehelCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment. Note that Icinga also has random failed checks (size of conntrack table, ferm, dpkg, ...) all with "Return code of 255 is out of bounds". The service (blazegraph) related checks are also failing, but that's expected, since the data isn't loaded yet.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, GehelCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. So I can see @Gehel asked: In T188045#4009517, @Gehel wrote: @Cmjohnson this looks like an issue with the physical connection. Could you try moving the cable to another port on the switch so that we can eliminate that possibility? In T188045#4014592, @Cmjohnson wrote: I replaced the ethernet cable and was able to login Debian GNU/Linux 8 auto-installed on Wed Sep 6 12:24:33 UTC 2017. cmjohnson@wdqs1004:~$ So it looks like chris swapped the cable, it worked, and the port wasn't changed? It is unlikely to be a bad EX4200 port (I can only recall this happening once in my time here) but it is possible. We are sill seeing bad ssh connections, and its likely ALL networking and SSH is just the most obvious symptom. I'd like to ask @Cmjohnson if the port was swapped on this? If not, can we go ahead and swap over the port?TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment. Bad news... I just finished reimaging wdqs1004, and I still have trouble. SSH session suddenly / randomly freezes. I can't see the same link flapping that @Dzahn saw in dmesg (at least not yet). @RobH / @Cmjohnson : any idea how to proceed?TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, GehelCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['wdqs1004.eqiad.wmnet'] and were ALL successful.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, ops-monitoring-botCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts: ['wdqs1004.eqiad.wmnet'] The log can be found in /var/log/wmf-auto-reimage/201803091444_gehel_17927.log.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, ops-monitoring-botCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
gerritbot added a comment. Change 417899 merged by Gehel: [operations/puppet@production] wdqs: migrate to stretch https://gerrit.wikimedia.org/r/417899TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, gerritbotCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
gerritbot added a comment. Change 417899 had a related patch set uploaded (by Gehel; owner: Gehel): [operations/puppet@production] wdqs: migrate to stretch https://gerrit.wikimedia.org/r/417899TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, gerritbotCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts: ['wdqs1004.eqiad.wmnet'] The log can be found in /var/log/wmf-auto-reimage/201803091357_gehel_7493.log.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobH, ops-monitoring-botCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
gerritbot added a comment. Change 417311 merged by Cmjohnson: [operations/puppet@production] dhcp mac address change wdqs1004 https://gerrit.wikimedia.org/r/417311TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, gerritbotCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
gerritbot added a comment. Change 417311 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson): [operations/puppet@production] dhcp mac address change wdqs1004 https://gerrit.wikimedia.org/r/417311TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, gerritbotCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Cmjohnson added a comment. The dell technician rescheduled for Thursday 8 March. Tentatively scheduled for 1800UTC.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: CmjohnsonCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Cmjohnson added a comment. yes march! I don't even know what month it is anymore..thxTASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: CmjohnsonCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Smalyshev added a comment. @Cmjohnson ITYM March 7?TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, SmalyshevCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Cmjohnson added a comment. The system board for this is scheduled to be changed out Wednesday 07FEB18TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: CmjohnsonCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Cmjohnson added a comment. a ticket has been placed with Dell You have successfully submitted request SR961706803.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: CmjohnsonCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Dzahn added a comment. from dmesg: [203721.906779] tg3 :02:00.0 eth0: Link is down [203725.760204] tg3 :02:00.0 eth0: Link is up at 1000 Mbps, full duplex [203725.760214] tg3 :02:00.0 eth0: Flow control is on for TX and on for RX [203725.760218] tg3 :02:00.0 eth0: EEE is disabled [203782.391437] tg3 :02:00.0 eth0: Link is down [203798.154227] tg3 :02:00.0 eth0: Link is up at 1000 Mbps, full duplex [203798.154236] tg3 :02:00.0 eth0: Flow control is on for TX and on for RX [203798.154239] tg3 :02:00.0 eth0: EEE is disabled [206269.350612] tg3 :02:00.0 eth0: Link is down [206289.517967] tg3 :02:00.0 eth0: Link is up at 1000 Mbps, full duplex [206289.517976] tg3 :02:00.0 eth0: Flow control is on for TX and on for RX [206289.517980] tg3 :02:00.0 eth0: EEE is disabledTASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, DzahnCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Dzahn added a comment. It's intermittent. Most of the time it works,but then it doesn't... [wdqs1004:~] $ host www.wikidata.org www.wikidata.org has address 208.80.154.224 www.wikidata.org has IPv6 address 2620:0:861:ed1a::1TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, DzahnCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Dzahn added a comment. Ack, i was able to connect to it and run ping for a while but .. it's like it's freezing sometimes and then continues. Also Gehel could confirm there is still an issue. :/ Can we do the "try other switch port" step next @Cmjohnson ?TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, DzahnCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Smalyshev added a comment. Still getting this: Mar 01 17:48:34 wdqs1004 bash[1382]: Exception in thread "main" java.lang.RuntimeException: java.net.UnknownHostException: www.wikidata.org looks like the problem is still there.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, SmalyshevCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-03-01T17:47:27Z] restarting wdqs-updater on wdqs1004 -T188045TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, StashbotCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-03-01T17:46:16Z] re-enabling icinga notifications for wdqs1004 services, ethernet cable has been replaced (T188045)TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, StashbotCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Dzahn added a comment. Thanks Chris! First i wasn't able to login, then i was and could start a puppet run. Let's see if Icinga clears up and stays green for a while.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, DzahnCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Cmjohnson added a comment. I replaced the ethernet cable and was able to login Debian GNU/Linux 8 auto-installed on Wed Sep 6 12:24:33 UTC 2017. cmjohnson@wdqs1004:~$TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: CmjohnsonCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment.
A few things to check (thanks for the pointers from my fellow ops):
is another server stealing its IP
check DNS -> nothing suspicious
check DHCP leases
check ARP caches -> sudo cumin -b200 -p 80 'A:eqiad' "arp -n | grep ^10.64.0.17\ | awk '{print \$3}'" shows only 18:66:da:ac:49:d5 which is the correct MAC of wdqs1004
check is network is downgraded to 10Mb -> ethtool say: Speed: 1000Mb/s, Duplex: Full
try moving wdqs1004 to another port on the switch
TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Smalyshev added a comment. If losing one server out of 4 is an issue One out of 3. We don't have a problem right now, but if we will have to take down another one - for maintenance or other issue - we will have it. If 1004 comes back within short timeframe, it's not a concern, but since it's now dead for almost a weak, I'm starting to think about options in case that (bringing it back to life) takes longer time.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, SmalyshevCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Joe added a comment. In T188045#4007098, @Smalyshev wrote: I wonder if it's possible to use one of the new servers we're getting in T187766 to restore full capacity if debugging what is going on with 1004 takes time. Would it be a good thing to do? If losing one server out of 4 is an issue, then we have a sizing problem for that cluster. Is that the case? I can't find evidence in the utilization metrics for those servers.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, JoeCc: Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment. There isn't much impact on response time or even load on the cluster at this point. So I would not worry too much yet. If we loose another node, this is going to be an issue, but let's not borrow trouble yet. Note that we have already received new servers in codfw, but wdqs1004 is in eqiad, so that does not help. We can re-discuss that once the new servers are available (they are configured exactly the same, so moving them between clusters should be trivial).TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-02-27T17:38:36Z] restarting wdqs-updater on wdqs1004 - T188045TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, StashbotCc: Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Gehel added a comment. Hardware diagnostics have completed with no error. It is now up again, catching on updates, but still depooled. I'll keep an eye on it, and if it looks stable and has catched up on updates, I'll repool it (and still keep an eye on it).TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-02-27T08:42:34Z] powercycling wdqs1004 - T188045TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, StashbotCc: Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
Dzahn added a comment. i ran puppet on all wdqs* via cumin after the merge and smalyshev confirmed the LDF server seems fine https://query.wikidata.org/bigdata/ldfTASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: DzahnCc: gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Gehel, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
gerritbot added a comment. Change 413485 merged by Dzahn: [operations/puppet@production] wdqs: move LDF endpoint to wdqs1005 since wdqs1004 is having hardware issues https://gerrit.wikimedia.org/r/413485TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Gehel, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
