[Wikidata-bugs] [Maniphest] T294297: Q2:(Need By: End of Q2) rack/setup/install wdqs20[09,10,11]
RobH assigned this task to Papaul. RobH moved this task from Backlog to Racking Tasks on the ops-codfw board. TASK DETAIL https://phabricator.wikimedia.org/T294297 WORKBOARD https://phabricator.wikimedia.org/project/board/955/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Papaul, RobH Cc: Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294297: Q2:(Need By: End of Q2) rack/setup/install wdqs20[09,10,11]
RobH added a parent task: Unknown Object (Task). TASK DETAIL https://phabricator.wikimedia.org/T294297 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: Aklapper, RobH, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294297: Q2:(Need By: End of Q2) rack/setup/install wdqs20[09,10,11]
RobH renamed this task from "(Need By: TBD) rack/setup/install wdqs20[09,10,11]" to "Q2:(Need By: End of Q2) rack/setup/install wdqs20[09,10,11]". TASK DETAIL https://phabricator.wikimedia.org/T294297 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: Aklapper, RobH, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294297: (Need By: TBD) rack/setup/install wdqs20[09,10,11]
RobH created this task. RobH added projects: DC-Ops, ops-codfw, Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION This task will track the racking, setup, and OS installation of wdqs20[09,10,11] Hostname / Racking / Installation Details - **Hostnames:** wdqs20[09,10,11] **Racking Proposal:** For context, we (before this ticket) currently have 8 codfw wdqs hosts, with 2 hosts in each row. After this ticket we will have **9 hosts total**. The 3 refreshes ideally should go in the same row as the servers they're replacing (`wdqs200[1-3]`). For reference: wdqs2001 - c5 wdqs2002 - d5 wdqs2003 - a5 Given the above, the 1 expansion host can go in any row, since we're already balanced across rows before this ticket. If for whatever reason the requested rows aren't available (ex: no 10G port in the requested row), let's just try to keep the rows as balanced as possible. For reference, here's every codfw wdqs host that //isn't// being refreshed: wdqs2004 - a6 wdqs2005 - b5 wdqs2006 - d4 wdqs2007 - b8 wdqs2008 - c6 **Networking/Subnet/VLAN/IP:** 10G, single port, production VLAN **Partitioning/Raid:** S/W RAID10 - partman: raid10-4dev.cfg **OS Distro:** Buster Per host setup checklist wdqs2009: [ ] - receive in system on #procurement <https://phabricator.wikimedia.org/tag/procurement/> task T291982 & in coupa [ ] - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned) [ ] - bios/drac/serial setup/testing [ ] - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox. [ ] - network port setup via netbox, run homer to commit [ ] - firmware update (idrac, bios, network, raid controller) [ ] - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). [ ] - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host [ ] - host state in netbox set to staged wdqs2010: [ ] - receive in system on #procurement <https://phabricator.wikimedia.org/tag/procurement/> task T291982 & in coupa [ ] - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned) [ ] - bios/drac/serial setup/testing [ ] - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox. [ ] - network port setup via netbox, run homer to commit [ ] - firmware update (idrac, bios, network, raid controller) [ ] - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). [ ] - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host [ ] - host state in netbox set to staged wdqs2011: [ ] - receive in system on #procurement <https://phabricator.wikimedia.org/tag/procurement/> task T291982 & in coupa [ ] - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned) [ ] - bios/drac/serial setup/testing [ ] - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox. [ ] - network port setup via netbox, run homer to commit [ ] - firmware update (idrac, bios, network, raid controller) [ ] - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). [ ] - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host [ ] - host state in netbox set to staged **Once the system(s) above have had all checkbox steps completed, this task can be resolved.** TASK DETAIL https://phabricator.wikimedia.org/T294297 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: Aklapper, RobH, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T263125: Check for errors on wdqs1009 disks
RobH added a comment. IRC update from my chat with @gehel: wdqs1009 is not in production, but a non user facing test server. We can do our hdparm testing of secure erasure without affecting users. If we nuke the server, it can simply be reimaged. When Chris is ready to swap the SSD, I'll want to login, use mdadm to pull SDB out of the sw raid10, then hdparm secure erase it. Chris can then swap the disk and I can attempt a sw raid rebuild. I'll document my steps and commands, so we can attempt to automate this in the future. TASK DETAIL https://phabricator.wikimedia.org/T263125 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, RobH Cc: RobH, wiki_willy, Gehel, EBernhardson, Aklapper, dcausse, lmata, CBogen, Akuckartz, darthmon_wmde, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T263125: Check for errors on wdqs1009 disks
RobH added a comment. I've pulled the TSR and submitted SR1038301301 for dispatching a replacement SSD to eqiad. This is now over to Chris for receipt and installation/return of defective SSD. Since the failing SSD is currently working, we need to attempt to securely erase it before sending back to Dell. The directions for using hdparm are documented on https://wikitech.wikimedia.org/wiki/Dc-operations/Securely_Erasing_Media and I've checked, wdqs1009 has hdparm installed within the OS. I'm not sure how well a system will handle hdparm secure erase on 1 of 4 disks in a sw raid10, but this seems like an excellent opportunity to find out! I'm willing to remotely assist as needed! I would recommend this system be depooled during the SSD swap, in case the secure erase (only lasts 3 minutes) causes undue loads. @gehel: Are there directions for [de/re]pooling of wdqs servers we can follow for this within #dc-ops <https://phabricator.wikimedia.org/tag/dc-ops/>? TASK DETAIL https://phabricator.wikimedia.org/T263125 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, RobH Cc: RobH, wiki_willy, Gehel, EBernhardson, Aklapper, dcausse, lmata, CBogen, Akuckartz, darthmon_wmde, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T221631: Dedicated servers on WMCS to test WDQS scalability strategy
RobH closed subtask T232654: eqiad: three clouvirt-wdqs servers for WDQS testing as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Andrew, RobH Cc: Iamamz3, GreenReaper, RazShuty, Krenair, Smalyshev, Andrew, Gehel, bd808, Aklapper, Phamhi, darthmon_wmde, Nandana, sietec, Bstorm, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Allthingsgo, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T221632: Storage capacity upgrade for WDQS
RobH added a comment. I'd like to note that I'm against the trend of our raid configurations becoming raid0 per node, and relying on horizontal node redundancy over disk redundancy. Losing a single disk in a raid0 node invalidates the entire node, making it a single point of urgency for our on-site engineers to address quickly. If we stick with some kind of actual disk redundancy, we lessen the urgency of those disk failures by an order of magnitude. TASK DETAIL https://phabricator.wikimedia.org/T221632 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel, RobH Cc: RobH, faidon, Smalyshev, Aklapper, Gehel, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202780: add SSDs to wdqs1003
RobH shifted this object from the Restricted Space space to the S1 Public space.RobH edited projects, added ops-eqiad; removed procurement. TASK DETAILhttps://phabricator.wikimedia.org/T202780EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: mark, faidon, Addshore, Smalyshev, Pswaby, RobH, Cmjohnson, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T202778: add ssds to wdqs2003
RobH renamed this task from "add more ssds to wdqs2003" to "add ssds to wdqs2003". TASK DETAILhttps://phabricator.wikimedia.org/T202778EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Aklapper, mark, faidon, Addshore, Smalyshev, RobH, Papaul, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T196485: WDQS diskspace is low
RobH added a comment. Please note all sub-tasks for additions to the wdqs cluster have been created and are linked off this task. Next steps are for coordination with @Gehel and onsites to add the disks. As the disks are hot swap addtions (not replacements), it shouldnt result in any downtime. However, its always best to be logged in and watching the host when swapping harware.TASK DETAILhttps://phabricator.wikimedia.org/T196485EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Cmjohnson, RobH, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T202779: add SSDs to wdqs100[45]
RobH created this task.RobH triaged this task as "Normal" priority.RobH added projects: Wikidata, Wikidata-Query-Service, Discovery-Wikidata-Query-Service-Sprint, Discovery, Operations, ops-eqiad. TASK DESCRIPTIONThis task will be used to coordinate a time between @Gehel and @Cmjohnson to add the new disks to wdqs100[45], purchased on T198658. Adding the disks (not replacing existing) shouldn't result in any downtime, but we should watch it carefully just to be certain. Then @Gehel will need to take over on manually extending the software raid/lvm onto the new disks to use the space.TASK DETAILhttps://phabricator.wikimedia.org/T202779EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: mark, faidon, Addshore, Smalyshev, RobH, Cmjohnson, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202778: add more ssds to wdqs2003
RobH added a project: ops-codfw. TASK DETAILhttps://phabricator.wikimedia.org/T202778EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Aklapper, mark, faidon, Addshore, Smalyshev, RobH, Papaul, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202777: add SSDs to wdqs200[12]
RobH added a project: ops-codfw. TASK DETAILhttps://phabricator.wikimedia.org/T202777EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Aklapper, mark, faidon, Addshore, Smalyshev, RobH, Papaul, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T202778: add more ssds to wdqs2003
RobH created this task.RobH triaged this task as "Normal" priority.RobH added projects: Wikidata, Wikidata-Query-Service, Discovery-Wikidata-Query-Service-Sprint, Discovery, Operations.Restricted Application added a subscriber: Aklapper. TASK DESCRIPTIONThis task will be used to coordinate a time between @Gehel and @Papaul to add the new disks to wdqs2003, purchased on T198654. Adding the disks (not replacing existing) shouldn't result in any downtime, but we should watch it carefully just to be certain. Then @Gehel will need to take over on manually extending the software raid/lvm onto the new disks to use the space.TASK DETAILhttps://phabricator.wikimedia.org/T202778EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Aklapper, mark, faidon, Addshore, Smalyshev, RobH, Papaul, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T202777: add SSDs to wdqs200[12]
RobH created this task.RobH triaged this task as "Normal" priority.RobH added projects: Wikidata, Wikidata-Query-Service, Discovery-Wikidata-Query-Service-Sprint, Discovery, Operations.Restricted Application added a subscriber: Aklapper. TASK DESCRIPTIONThis task will be used to coordinate a time between @Gehel and @Papaul to add the new disks to wdqs200[12], purchased on T198657 Adding the disks (not replacing existing) shouldn't result in any downtime, but we should watch it carefully just to be certain. Then @Gehel will need to take over on manually extending the software raid/lvm onto the new disks to use the space.TASK DETAILhttps://phabricator.wikimedia.org/T202777EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Aklapper, mark, faidon, Addshore, Smalyshev, RobH, Papaul, Gehel, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T196485: WDQS diskspace is low
RobH closed subtask T186526: codfw/eqiad: WDQS storage increase as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T196485EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T196485: WDQS diskspace is low
RobH added a comment. In T196485#4258384, @Gehel wrote: We have a "sleeping" task to order new disks: T186526 This has been resolved, as there are now 4 sub-tasks off it for the ordering of the ssds for each of the hosts listed in this task. As they are #procurement tasks, they are non-public (thus my update here.)TASK DETAILhttps://phabricator.wikimedia.org/T196485EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T196485: WDQS diskspace is low
RobH closed subtask T186526: codfw/eqiad: WDQS storage increase as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T196485EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T195797: Stas needs root access on WDQS test cluster
RobH closed this task as "Resolved".RobH claimed this task.RobH added a comment. These patches have been merged, and should be live shortly on hosts!TASK DETAILhttps://phabricator.wikimedia.org/T195797EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, MoritzMuehlenhoff, gerritbot, Gehel, Aklapper, Smalyshev, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, MarcoAurelio, aude, Tobias1984, Manybubbles, mark, JanZerebecki, faidon, Mbch331, Rxy, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T195797: Stas needs root access on WDQS test cluster
RobH added subscribers: MoritzMuehlenhoff, RobH.RobH added a comment. Please note this was approved in today's SRE team meeting. I'll go ahead and start merging @MoritzMuehlenhoff's patches.TASK DETAILhttps://phabricator.wikimedia.org/T195797EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, MoritzMuehlenhoff, gerritbot, Gehel, Aklapper, Smalyshev, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, MarcoAurelio, aude, Tobias1984, Manybubbles, mark, JanZerebecki, faidon, Mbch331, Rxy, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T194184: rack/setup/install wdqs10[09|10].eqiad.wmnet
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...Racking Proposal: Currently, there are 7 other wdqs systems (wdqs1002-1008). Of those 7, only 1 (wdqs2002) is out of warranty. As such, we should attempt to place the two new wdqs systems in racks other than the existing ones. However, if there has to be overlap of one, overlap with wdqs2002. 7 racks is a LOT of racks to try to avoid usingDiscussion with @gehel via irc notes these two new systems will be in their own wdqs cluster, so minimal overlap may be expectedunrelated to the existing one. (TSo these are 1G NIC systemsshould just be in different rows than one another, so no reason to put in a future 10G rack when avoidable.)but have no relation to the other 7 wdqs systems. The locations of the other wdqs servers are (in order of wdqs1002-1009): c7, a4, a6, d3, a1, b1, d1. wdqs1009:...TASK DETAILhttps://phabricator.wikimedia.org/T194184EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Gehel, Smalyshev, Aklapper, Cmjohnson, faidon, RobH, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T194184: rack/setup/install wdqs10[09|10].eqiad.wmnet
RobH created this task.RobH triaged this task as "Normal" priority.RobH added projects: Wikidata-Query-Service, Discovery-Search, Operations, Wikidata, Discovery, Epic, Discovery-Wikidata-Query-Service-Sprint, ops-eqiad. TASK DESCRIPTIONThis task will track the racking, setup, and installation of two new wdqs systems, wdqs1009 & wdqs1010. Racking Proposal: Currently, there are 7 other wdqs systems (wdqs1002-1008). Of those 7, only 1 (wdqs2002) is out of warranty. As such, we should attempt to place the two new wdqs systems in racks other than the existing ones. However, if there has to be overlap of one, overlap with wdqs2002. 7 racks is a LOT of racks to try to avoid using, so minimal overlap may be expected. (These are 1G NIC systems, so no reason to put in a future 10G rack when avoidable.) The locations of the other wdqs servers are (in order of wdqs1002-1009): c7, a4, a6, d3, a1, b1, d1. wdqs1009: - receive in system on #procurement task T186713 - rack system with proposed racking plan (see above) & update racktables (include all system info plus location) - bios/drac/serial setup/testing - mgmt dns entries added for both asset tag and hostname - network port setup (description, enable, private vlan) end on-site specific steps - production dns entries added (private vlan) - operations/puppet update (install_server at minimum, other files if possible) - OS installation - puppet accept/initial run - handoff for service implementation wdqs1010: - receive in system on #procurement task T186713 - rack system with proposed racking plan (see above) & update racktables (include all system info plus location) - bios/drac/serial setup/testing - mgmt dns entries added for both asset tag and hostname - network port setup (description, enable, private vlan) end on-site specific steps - production dns entries added (private vlan) - operations/puppet update (install_server at minimum, other files if possible) - OS installation - puppet accept/initial run - handoff for service implementation TASK DETAILhttps://phabricator.wikimedia.org/T194184EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Gehel, Smalyshev, Aklapper, Cmjohnson, faidon, RobH, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T193473: Add HTTPS support to wdqs-internal service
RobH triaged this task as "Normal" priority.RobH added a comment. As part of SRE clinic duty, I'm reviewing all unassigned, needs triage tasks in #operations and attempting to review if any are critical, or if they are normal priority. This task appears to be normal priority, and I have set it such. If any one on this task disagrees, please comment and correct. Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T193473EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Smalyshev, Gehel, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T193733: Move dispatching of wikidata to a dedicated node
RobH triaged this task as "Normal" priority.RobH added a comment. As part of SRE clinic duty, I'm reviewing all unassigned, needs triage tasks in #operations and attempting to review if any are critical, or if they are normal priority. This task appears to be normal priority, and I have set it such. If any one on this task disagrees, please comment and correct. Anything with a high priority or above typically requires response ahead of other items, so please ensure you have supporting documentation on why those priorities should be used. Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T193733EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, hoo, Aklapper, Lydia_Pintscher, gerritbot, Sjoerddebruin, MisterSynergy, Lea_Lacroix_WMDE, Ladsgroup, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T192452: Find out how many external uses the wbcheckconstraints API action has
RobH closed subtask T190415: Requesting access to stats machines for Lucas Werkmeister as "Declined". TASK DETAILhttps://phabricator.wikimedia.org/T192452EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Lucas_Werkmeister_WMDE, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Agabi10, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. In T188045#4045150, @Platonides wrote: Well, if the server itself is needed, it will be doing its work with a different IP address than the one of wdqs1004, since it would have been suffering the same connection issues as wdqs1004 (just the other half time). The other server was decommissioned, not needed! So no worries.TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Platonides, MoritzMuehlenhoff, TerraCodes, faidon, Liuxinyu970226, ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, ayounsi, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, mark, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. Ok, I misparsed all of that. So next steps: Chris traces out and sees what is connected to ge-4/0/18. It somehow has the same IP address as wdqs1004 and is just sitting there. If we can, we shut down that server. If we cannot, we fix dns so its pointing correctly. If wdqs1004 needs a new ip, update dns and reimage. TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: MoritzMuehlenhoff, TerraCodes, faidon, Liuxinyu970226, ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, ayounsi, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, mark, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T188045: wdqs1004 broken
RobH added a comment. So I can see @Gehel asked: In T188045#4009517, @Gehel wrote: @Cmjohnson this looks like an issue with the physical connection. Could you try moving the cable to another port on the switch so that we can eliminate that possibility? In T188045#4014592, @Cmjohnson wrote: I replaced the ethernet cable and was able to login Debian GNU/Linux 8 auto-installed on Wed Sep 6 12:24:33 UTC 2017. cmjohnson@wdqs1004:~$ So it looks like chris swapped the cable, it worked, and the port wasn't changed? It is unlikely to be a bad EX4200 port (I can only recall this happening once in my time here) but it is possible. We are sill seeing bad ssh connections, and its likely ALL networking and SSH is just the most obvious symptom. I'd like to ask @Cmjohnson if the port was swapped on this? If not, can we go ahead and swap over the port?TASK DETAILhttps://phabricator.wikimedia.org/T188045EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: ops-monitoring-bot, Joe, RobH, Stashbot, Gehel, Cmjohnson, Papaul, gerritbot, elukey, Smalyshev, Aklapper, Dzahn, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Zppix, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T186526: Increase storage space for Wikidata Query Service
RobH added a comment. Actually, each of the wdqs systems is mildly different, and may have different ssds and capacities. I'm moving this task into S4 #procurement, since it'll include pricing discussion.TASK DETAILhttps://phabricator.wikimedia.org/T186526EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Addshore, Smalyshev, Gehel, RobH, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T186526: Increase storage space for Wikidata Query Service
RobH claimed this task.RobH added a comment. I'll request quotation from the vendor for adding SSDS. I'll make a sub-task in #procurement, since it will contain pricing info.TASK DETAILhttps://phabricator.wikimedia.org/T186526EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Addshore, Smalyshev, Gehel, RobH, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T186713: Setup a WDQS test cluster on real hardware
RobH triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T186713EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Aklapper, EBjune, Smalyshev, Gehel, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T186526: Increase storage space for Wikidata Query Service
RobH added a comment. I'd advise we purchase the SSDs from the system manufacturers, so they would be covered under the system warranty. I can get pricing, but it'll need to go in an S4 task (since pricing isn't public info.) I don't recommend using third party SSDs, since they won't be covered under the system warranty, and its difficult to keep the new and old SSDs sorted that way within the same system.TASK DETAILhttps://phabricator.wikimedia.org/T186526EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, RobHCc: Addshore, Smalyshev, Gehel, RobH, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T182991: New WDQS clusters eqiad + codfw
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...All quotations will be on the sub-task for the codfw system purchase, task T183201. NOT on this task. Please see sub-task T183201 for all followup. Please seeOnce the codfw quotation is agreed upon, @robh will make another sub-task for all followupthe eqiad purchase. TASK DETAILhttps://phabricator.wikimedia.org/T182991EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Gehel, EBjune, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, merbst, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Blocker] T178492: Create a more controlled WDQS cluster
RobH changed the status of subtask T182991: New WDQS clusters eqiad + codfw from "Open" to "Stalled". TASK DETAILhttps://phabricator.wikimedia.org/T178492EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Daniel_Mietchen, BBlack, Lydia_Pintscher, Volans, mobrovac, Jonas, Aklapper, debt, Smalyshev, Gehel, Lahi, PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, SandraF_WMF, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, Avner, FloNight, Xmlizer, Susannaanas, Eevans, Aschroet, Jane023, jkroll, Hardikj, Wikidata-bugs, Jdouglas, PKM, Base, matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Fabrice_Florin, Raymond, Steinsplitter, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Status] T182991: New WDQS clusters eqiad + codfw
RobH changed the task status from "Open" to "Stalled".RobH triaged this task as "Normal" priority.RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...@EBjune can you confirm we have a budget for this? == Details for hw-request/procurement == This task was created and currently lives in the public space, in #hw-requests. As such, it cannot have quote/totals posted directly into this task. All price discussion will have to take place in the #procurement sub-tasks. Since this is for two sites, there will have to be two #procurement tasks (one task per order.) I'll (@robh) will create them and link them off this task. I'll only process 1 of the 2 right away, and get a working quote/specification with the vendor. The last order of wdqs systems was on T166780, however the quote from that task CANNOT be linked in this public task. Rather, I'll just list off the hardware of the past system here, but the sub-task for a codfw quote will follow and have full pricing details: wdqs100[56] are the most recent purchase of wdqs systems. * Dell Poweredge R430 ** Please note that the R430 only allows 4 dimm slots dedicated to CPU2 (there are 12 slots total, but 8 go to CPU 1 only). This config has 4 total dimms, 2 going to CPU2. * Dual [[ https://ark.intel.com/products/92986/Intel-Xeon-Processor-E5-2620-v4-20M-Cache-2_10-GHz | Intel Xeon E5-2620 v4 ]] (2.10GHz 8 cores) * 128GB RAM * no hardware raid, onboard sata controller only * Dual Intel S3520 800 GB SSD * 1 Gbit (onboard) network card (4 ports) * drac enterprise * dual psu (3' c13/c14 power cables) * no bezel, no optical * rails without cable management All quotations will be on the sub-task for the codfw system purchase, NOT on this task. Please see sub-task for all followup. TASK DETAILhttps://phabricator.wikimedia.org/T182991EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Gehel, EBjune, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, merbst, Avner, Zppix, Jonas, FloNight, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T178271: Allow Kirk and Martijn (JClarity) access to our WDQS production servers
RobH closed this task as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T178271EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, RobHCc: MoritzMuehlenhoff, RStallman-legalteam, Karianna, debt, EBjune, Gehel, Aklapper, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, merbst, Avner, Zppix, Jdrewniak, Jonas, FloNight, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, mark, JanZerebecki, faidon, Mbch331, Rxy, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH added a comment. I've assigned this to @Gehel as both hosts are now online and ready for service implementation.TASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, RobHCc: gerritbot, PokestarFan, Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reassigned] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH reassigned this task from RobH to Gehel.RobH removed projects: Patch-For-Review, ops-eqiad.RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...[x] - handoff for service implementation...[x] - OS installation [x] - puppet/salt accept/initial run [x] - handoff for service implementationTASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, RobHCc: gerritbot, PokestarFan, Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...[x] - operations/puppet update (install_server at minimum, other files if possible) [x] - OS installation [x] - puppet/salt accept/initial run...[x] - operations/puppet update (install_server at minimum, other files if possible)...TASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: gerritbot, PokestarFan, Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Zoranzoki21, merbst, Avner, Lewizho99, Zppix, Maathavan, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH claimed this task. TASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: gerritbot, PokestarFan, Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Zoranzoki21, merbst, Avner, Lewizho99, Zppix, Maathavan, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T172798: allow wdqs-admins to pool / depool wdqs servers
RobH added a comment. This access request was reviewed and approved in today's (2017-08-21) operations team meeting. Giving these admins the rights to depool and repool individual systems within their purview was granted.TASK DETAILhttps://phabricator.wikimedia.org/T172798EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, RobHCc: RobH, gerritbot, hoo, Aklapper, Smalyshev, Gehel, PokestarFan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Zppix, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, mark, JanZerebecki, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Reassigned] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH reassigned this task from RobH to Cmjohnson.RobH added a comment. So wdqs1004 shows a link, and I'll proceed to install it. I can login to mgmt on wdqs1005, but its actual network port shows link down: Interface Admin Link Description ge-3/0/3updown wdqs1005 @Cmjohnson: I'm assigning this back to you for you to investigate the link of wdqs1005. Once it is working on that port, please assign this back to me. I'll continue to work on the installation of wdqs1004 and will continue on wdqs1005 after network link repair. Thanks!TASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: PokestarFan, Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...Racking Proposal: These shouldn't be in any racks with existing wdqs systems (if possible), so do NOT place in a4 (wdqs1001). Place in any other 1GbE rack, since these use internal subnets. No two wdqs systems should share a rack, and try to put the three new ones in rows B, C, and DTASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...Racking Proposal: These shouldn't be in any racks with existing wdqs systems (if possible), so do NOT place in a4 (wdqs1001). Place in any other 1GbE rack, since these use internal subnets. No two wdqs systems should share a rackTASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION...Racking Proposal: These shouldn't be in any racks with existing wdqs systems (if possible), so do NOT place in a4 (wdqs1001). Place in any other 1GbE rack, so do NOT place in d3since these use internal subnets. Please note that it is likely that d3 (wdqs1001), c7(wdqs1002) will be decommissioned after these are fully online. So if there isn't 1GbE rack space outside of D3 or c7, & a4those two racks can be used. Place in any other 1GbE rackIf there is space outside them, sinplace these use internal subnetsese systems elsewhere to eliminate potential redundancy issues until the old ones are decommissionedTASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Aklapper, Gehel, Smalyshev, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T171210: rack/setup/install wdqs100[45].eqiad.wmnet
RobH created this task.RobH added projects: Wikidata-Query-Service, Operations, Wikidata, Discovery, Discovery-Search (Current work).Herald added a subscriber: Aklapper. TASK DESCRIPTIONThis task will track the racking and setup of wdqs100[45].eqiad.wmnet. Racking Proposal: These shouldn't be in any racks with existing wdqs systems, so do NOT place in d3, c7, & a4. Place in any other 1GbE rack, since these use internal subnets. wdqs1004: - receive in system on #procurement task T166780 - rack system with proposed racking plan (see above) & update racktables (include all system info plus location) - bios/drac/serial setup/testing - mgmt dns entries added for both asset tag and hostname - network port setup (description, enable, internal vlan) end on-site specific steps - production dns entries added (internal vlan) - operations/puppet update (install_server at minimum, other files if possible) - OS installation - puppet/salt accept/initial run - handoff for service implementation wdqs1005: - receive in system on #procurement task T166780 - rack system with proposed racking plan (see above) & update racktables (include all system info plus location) - bios/drac/serial setup/testing - mgmt dns entries added for both asset tag and hostname - network port setup (description, enable, internal vlan) end on-site specific steps - production dns entries added (internal vlan) - operations/puppet update (install_server at minimum, other files if possible) - OS installation - puppet/salt accept/initial run - handoff for service implementation TASK DETAILhttps://phabricator.wikimedia.org/T171210EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Aklapper, Gehel, Smalyshev, faidon, Pswaby, Cmjohnson, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jdrewniak, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T166780: Replace wdqs100[12] servers as they are getting old
RobH added a comment. As these have SSDs, their lead time will extend past this fiscal year, and into next. I'll get quotes started.TASK DETAILhttps://phabricator.wikimedia.org/T166780EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Smalyshev, Gehel, RobH, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T148747: Estimate hardware requirements for WDQS upgrade
RobH closed this task as "Resolved".RobH added a comment. As both wdqs expansion systems have been ordered and have setup tasks, I'm resolving this hardware-request.TASK DETAILhttps://phabricator.wikimedia.org/T148747EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Deskana, RobHCc: OakleyAlways1, RobH, mark, K4-713, Sjoerddebruin, Gehel, Aklapper, Deskana, Smalyshev, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T148747: Estimate hardware requirements for WDQS upgrade
RobH reopened subtask Unknown Object (Task) as "Open". TASK DETAILhttps://phabricator.wikimedia.org/T148747EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Deskana, RobHCc: OakleyAlways1, RobH, mark, K4-713, Sjoerddebruin, Gehel, Aklapper, Deskana, Smalyshev, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T152644: rack/setup/install wdqs2003
RobH edited the task description. (Show Details) EDIT DETAILS...[x] - install_module update (same partitioning as wdqs200[12] if possible)...TASK DETAILhttps://phabricator.wikimedia.org/T152644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Southparkfan, Smalyshev, Aklapper, Deskana, mark, emailbot, Papaul, Cmjohnson, RobH, Gehel, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T152644: rack/setup/install wdqs2003
RobH added a comment. All of the install_module and dns changes that @Papaul put in gerrit for review have been merged live on cluster, @Papaul can continue with the installation.TASK DETAILhttps://phabricator.wikimedia.org/T152644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Southparkfan, Smalyshev, Aklapper, Deskana, mark, emailbot, Papaul, Cmjohnson, RobH, Gehel, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T152644: rack/setup/install wdqs2003
RobH closed subtask T153094: wdqs2003 switch port configuration as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T152644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Southparkfan, Smalyshev, Aklapper, Deskana, mark, emailbot, Papaul, Cmjohnson, RobH, Gehel, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T153094: wdqs2003 switch port configuration
RobH closed this task as "Resolved".RobH added a comment. network port enabled, description set, and put in the internal vlan.TASK DETAILhttps://phabricator.wikimedia.org/T153094EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Gehel, RobH, Cmjohnson, emailbot, mark, Deskana, Aklapper, Smalyshev, Southparkfan, Papaul, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, akosiaris, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T152644: rack/setup/install wdqs2003
RobH created this task.RobH added projects: Discovery, Wikidata-Query-Service, Wikidata, Operations, ops-codfw.Herald added a subscriber: Southparkfan. TASK DESCRIPTIONThis task will track the racking and setup of the new system host ordered on T149351 as wdqs2003. wdqs2001 is racked in c5, and wdqs2002 is racked in d5. wdqs2003 should rack in a different rack than the first two, a5-codfw seems less populated than b5, and both are misc racks, so I'd (@RobH) suggest there. - receive system, asset tag, attach packing slip to parent task T149351 - rack in a5-codfw - create sub-task (or update this task) with port info for network - setup mgmt dns (hostname and asset tag) and production dns (hostname in private vlan) - install_module update (same partitioning as wdqs200[12] if possible) - install OS - handoff to @Gehel for service implementation TASK DETAILhttps://phabricator.wikimedia.org/T152644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Papaul, RobHCc: Southparkfan, Smalyshev, Aklapper, Deskana, mark, emailbot, Papaul, Cmjohnson, RobH, Gehel, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T152643: rack/setup/install wdqs1003
RobH created this task.RobH added projects: Discovery, Wikidata-Query-Service, Wikidata, Operations, ops-eqiad.Herald added a subscriber: Southparkfan. TASK DESCRIPTIONThis task will track the racking and setup of the new system host ordered on T149351 as wdqs1003. wdqs1001 is racked in d3, and wdqs1002 is racked in c7. wdqs2003 should rack in a different rack than the first two. - receive system, asset tag, attach packing slip to parent task T149351 - rack in a different rack than wdqs100[12] - network port description and vlan update - setup mgmt dns (hostname and asset tag) and production dns (hostname in private vlan) - install_module update (same partitioning as wdqs100[12] if possible) - install OS - handoff to @Gehel for service implementation TASK DETAILhttps://phabricator.wikimedia.org/T152643EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Cmjohnson, RobHCc: Southparkfan, Smalyshev, Aklapper, Deskana, mark, emailbot, Papaul, Cmjohnson, RobH, Gehel, Th3d3v1ls, EBjune, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Status] T148747: Estimate hardware requirements for WDQS upgrade
RobH added subscribers: K4-713, mark, RobH.RobH changed the task status from "Open" to "Stalled".RobH added a comment. I received an IRC notice from Mark to start working on this, from an out of band coversation between @mark and @K4-713. I've created the sub-tasks in the #procurement S4 space, due to private pricing data. All quotes will need to be reviewed on sub-task T149351. As it seems that is all the info needed on this, I'm going to set it to stalled, while quote selection is in progress on the sub-task.TASK DETAILhttps://phabricator.wikimedia.org/T148747EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Deskana, RobHCc: RobH, mark, K4-713, Sjoerddebruin, Gehel, Aklapper, Deskana, Smalyshev, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Block] T148747: Estimate hardware requirements for WDQS upgrade
RobH created subtask T149351: codfw/eqiad: (1 each) wdqs cluster expansion. TASK DETAILhttps://phabricator.wikimedia.org/T148747EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Deskana, RobHCc: Sjoerddebruin, Gehel, Aklapper, Deskana, Smalyshev, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T149351: codfw/eqiad: (1 each) wdqs cluster expansion
RobH created this task.RobH added projects: Discovery, Wikidata-Query-Service, Wikidata, Operations, procurement. TASK DESCRIPTIONThis task will track the quote(s) and order(s) for expanding the wdqs cluster in both codfw and eqiad. These systems were most recently ordered for codfw on T139482. At that time, they used quote F4292858: Quote_US_PC_SC_1024178767306.1_2016-07-20.pdf. original #hardware-requests task data In order to be able to integrate WDQS with more on-wiki data sources and visualisations, @Smalyshev informs me that some scaling up of the hardware is required. He estimated that we needed one extra node. This task is for us to put that estimate together so it can be passed on to Katie Horn. Hardware estimate: Hardware should be similar to the current wdqs200[12] configuration: CPU: dual Intel(R) Xeon(R) CPU E5-2620 v3 Disk: 800GB raw raided space SSD (we could make do with 400GB) RAM: 96GB number of servers: eqiad: 1 codfw: 1 (we might be able to do without the codfw server in the short term if that is an issue)TASK DETAILhttps://phabricator.wikimedia.org/T149351EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Deskana, Aklapper, Gehel, Sjoerddebruin, RobH, Smalyshev, mschwarzer, Avner, Zppix, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T138637: codfw: (2) wqds200[12] systems
RobH closed this task as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T138637EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: Sjoerddebruin, hoo, RobH, Aklapper, Joe, Gehel, Zppix, Smalyshev, mschwarzer, Avner, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, faidon, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T124862: Deploy WDQS nodes on codfw
RobH closed subtask T138637: codfw: (2) wqds200[12] systems as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T124862EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: hoo, Gehel, Tfinc, Deskana, Joe, EBernhardson, Aklapper, Smalyshev, mschwarzer, Avner, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Manybubbles, ArielGlenn, faidon, Mbch331, Jay8g, greg, mmodell___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T138637: codfw: (2) wqds200[12] systems
RobH added a subtask: Unknown Object (Task). TASK DETAILhttps://phabricator.wikimedia.org/T138637EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: hoo, RobH, Aklapper, Joe, Gehel, Zppix, Smalyshev, Avner, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T138637: codfw: (2) wqds200[12] systems
RobH changed the title from "Hardware request for codfw WDQS server" to "codfw: (2) wqds200[12] systems".RobH claimed this task. TASK DETAILhttps://phabricator.wikimedia.org/T138637EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: hoo, RobH, Aklapper, Joe, Gehel, Zppix, Smalyshev, Avner, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T138637: Hardware request for codfw WDQS server
RobH added a comment. wdqs100[12] have the following hardware: Dell PowerEdge R420 Dual Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz (6 cores) 96GB RAM SSDs in a hw raid10 to total 500GB space currently 4 * 300GB We have no spares in any site to match this specification right now. I'll create a S4 #procurement task for the quoting of servers.TASK DETAILhttps://phabricator.wikimedia.org/T138637EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: hoo, RobH, Aklapper, Joe, Gehel, Zppix, Smalyshev, Avner, debt, D3r1ck01, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T134017: Create Wikipedia Jamaican
RobH added a blocking task: T135029: adywiki and jamwiki are missing the associated *_p databases with appropriate views. TASK DETAILhttps://phabricator.wikimedia.org/T134017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Lydia_Pintscher, StevenJ81, hoo, aude, Multichill, VIGNERON, Nikki, chasemp, yuvipanda, Andrew, Volans, Stashbot, MZMcBride, zhuyifei1999, Meno25, Dereckson, Mtherwjs, Glaisher, Liuxinyu970226, Dzahn, Ebe123, gerritbot, jcrespo, Krenair, SPQRobin, Aklapper, Matanya, JEumerus, MF-Warburg, Lewizho99, Maathavan, Urbanecm, D3r1ck01, Tulsi_Bhagat, Izno, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T134017: Create Wikipedia Jamaican
RobH added a comment. It sounds like a database script, and therefore falls to @jcrespo? (I don't want to leave this sitting with no attention, so just checking if he would be best.) I've pinged him about this via IRC.TASK DETAILhttps://phabricator.wikimedia.org/T134017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: RobHCc: RobH, Lydia_Pintscher, StevenJ81, hoo, aude, Multichill, VIGNERON, Nikki, chasemp, yuvipanda, Andrew, Volans, Stashbot, MZMcBride, zhuyifei1999, Meno25, Dereckson, Mtherwjs, Glaisher, Liuxinyu970226, Dzahn, Ebe123, gerritbot, jcrespo, Krenair, SPQRobin, Aklapper, Matanya, JEumerus, MF-Warburg, Lewizho99, Maathavan, Urbanecm, D3r1ck01, Tulsi_Bhagat, Izno, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T120712: install two Intel 320 Series SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002
RobH added a comment. I imagine the smaller capacity disks were installed by accident, as we have on shelf spares of that model SSD in both capacities. TASK DETAIL https://phabricator.wikimedia.org/T120712 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, RobH Cc: Gehel, Southparkfan, Cmjohnson, Smalyshev, StudiesWorld, Joe, Aklapper, hoo, fgiunchedi, mark, RobH, Avner, debt, D3r1ck01, FloNight, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T120714: implement wdqs1001/1002 disk upgrades (extend lvm)
RobH added a comment. That works for me, and I'll be around that entire week to assist on the operations end of things. Would we be able to include the re-image in that downtime window, or are we not certain about that quite yet? TASK DETAIL https://phabricator.wikimedia.org/T120714 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, Cmjohnson, debt, TerraCodes, Gehel, D3r1ck01, FloNight, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T120714: implement wdqs1001/1002 disk upgrades (extend lvm)
RobH added a comment. Barring any issues during the reimagine (as long as nothing breaks) it can typically be reinstalled in around an hour. This includes enabling the new disks, adding/updating the partman recipe (which we can do the bulk of before downtime), reinstalling the OS, and the initial post OS puppet runs. This doesn't include service re-implementation or the transfer of any data to or from the machine before or after reinstallation. TASK DETAIL https://phabricator.wikimedia.org/T120714 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, Cmjohnson, debt, TerraCodes, Gehel, D3r1ck01, FloNight, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T120714: implement wdqs1001/1002 disk upgrades (extend lvm)
RobH added a comment. At first glance, neither of these is showing the extra disks. It turns out these have the H310 controller, which means we need to enable the disks in the controller. If this was a typical raid controller, I just would do this while they are in service. The H310 is not one however, which is why we run it in bypass mode. Due to my fear of this system, I rather we schedule a potential downtime window for attempting to enable the disks and extending the LVM manually. Please note that manually extending the LVM is non-ideal compared to re-image (which was proposed to happen before or around now anyhow if possible). The benefits of the re-image is we would apply a new automatic partitioning recipe to these systems in production use. If a system crashes and requires a replacement/re-installation in the future, we have a known good recipe. Right now we had a known good recipe, but then we'll manually extend the other disks into it. This is non-ideal in long term use, as we cannot automatically re-create the setup on another system. Please advise if we can schedule potential downtime on one of the hosts to tinker with the H310 to enable the disks. Alternatively, we can take one offline and reimage it entirely, which would give the above listed benefits. TASK DETAIL https://phabricator.wikimedia.org/T120714 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, Cmjohnson, debt, TerraCodes, Gehel, D3r1ck01, FloNight, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T114443: EventBus MVP
RobH closed blocking task T114191: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, RobH Cc: Glaisher, Milimetric, RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle, EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Eevans, Ottomata, Matanya, Aklapper, jkroll, Hardikj, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, jayvdb, daniel, Mbch331, Jay8g, Ltrlg, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH added a comment. Well, we have @Mark's approval to use the on-site spares for this (as we have plenty.) Now the only question is how best to implement. We could do as @Joe suggests and add the two additional disks as raid1 and extend the LVM onto them. Ideally, we setup future installs to be raid10 across all 4 disks. If we cannot afford any reimage time now, we can still do that. As @fgiunchedi points out, we'll want to ensure partman is updated so when reinstall does roll around, its not broken. @Smalyshev: If we can afford the downtime, I think its better longterm to reimage them with raid10 and a new lvm. If we cannot afford the downtime, then we can do as @joe suggests and simply raid1 add the new disks and extend the LVM. Please advise (both @smalyshev and anyone else with knowledge on the subject) I'll create a sub-task to pull and add the additional disks, but with a note not to install them without checking first. (Seating them shouldn't cause any issues, but we should be certain and actively monitoring the system during this.) TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH added a comment. My issue is if we toss these in as a raid1 and extend the LVM, that is NOT how we would typically install it from scratch. So when these systems eventually upgrade and reinstall the distro, it'll involve changing the raid setup and potentially mount points, etc... I prefer to reimage and tackle these issues at the time of the disk upgrade, rather than push down the road. TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH added a comment. That sounds reasonable to me. I'll create the sub-tasks for the on-site installation of the disks and implementation of both raid1 and space + future reimage. TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T120712: install two Intel 320 Series SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002
RobH created this task. RobH assigned this task to Cmjohnson. RobH added subscribers: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, Cmjohnson. RobH added projects: Discovery, Wikidata-Query-Service, Wikidata, operations, ops-eqiad. TASK DESCRIPTION @cmjohnson, We'll be upgrading the space in wdqs1001 & wdqs1002. This task will track the installation of these disks into the systems. Each system needs two Intel 320 Series SSDSA2CW300G3 2.5" 300GB installed. These should be able to hot-plug in, but please have ssh open and monitoring the system when disks are installed to ensure they do not halt/freeze the system. There is no preference in order of upgrade (can upgrade 1001 or 1002 first.) Previously IRC discussion showed that eqiad has spare disk caddies for use in the R430 SFF chassis. Once these are installed, the sub-task can be worked to extend the lvm/partition. TASK DETAIL https://phabricator.wikimedia.org/T120712 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, RobH Cc: Cmjohnson, Smalyshev, StudiesWorld, Joe, Aklapper, hoo, fgiunchedi, mark, RobH, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH closed this task as "Resolved". RobH added a comment. Resolving this task as the onsite task now exists. TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T120714: implement wdqs1001/1002 disk upgrades (extend lvm)
RobH created this task. RobH added subscribers: Cmjohnson, Smalyshev, StudiesWorld, Joe, Aklapper, hoo, fgiunchedi, mark, RobH. RobH added projects: operations, Wikidata, Wikidata-Query-Service, Discovery. TASK DESCRIPTION Once T120712 is completed, the newly installed disks will need to be setup into a software raid1 and have the lvm extended onto these disks for increased storage space. Discussion on past tasks T119579 with @Smalyshev shows that we need to extend space in place and not reimage these systems at this time. Once the holidays are past and everyone on his team is back from holiday/vacation, we can then look at reimaging these one at a time (in January.) So this task can track initial implementation, and then stall until January for reimage. TASK DETAIL https://phabricator.wikimedia.org/T120714 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, Cmjohnson, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Assigned] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH added subscribers: mark, RobH. RobH assigned this task to Smalyshev. RobH added a comment. Ok some working notes: Both wdqs1001/wdqs1002 are Dell Poweredge R420s that have space for 8 total SFF (2.5") disks. We presently have dual 300GB installed into each. robh@wdqs1002:~$ sudo megacli -PDList -aALL Adapter #0 Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 0 WWN: 500151795961c7a4 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 279.460 GB [0x22eec130 Sectors] Non Coerced Size: 278.960 GB [0x22dec130 Sectors] Coerced Size: 278.875 GB [0x22dc Sectors] Sector Size: 0 Firmware state: Online, Spun Up Device Firmware Level: 0302 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x443322110400 Connected Port Number: 0(path0) Inquiry Data: CVPR130402SH300EGN INTEL SSDSA2CW300G3 4PC10302 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 3.0Gb/s Link Speed: 3.0Gb/s Media Type: Solid State Device Drive: Not Certified Drive Temperature : N/A PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 3.0Gb/s Drive has flagged a S.M.A.R.T alert : No Enclosure Device ID: 32 Slot Number: 1 Drive's position: DiskGroup: 0, Span: 0, Arm: 1 Enclosure position: 1 Device Id: 1 WWN: 5001517bb2844a0b Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA Raw Size: 279.460 GB [0x22eec130 Sectors] Non Coerced Size: 278.960 GB [0x22dec130 Sectors] Coerced Size: 278.875 GB [0x22dc Sectors] Sector Size: 0 Firmware state: Online, Spun Up Device Firmware Level: 0362 Shield Counter: 0 Successful diagnostics completion on : N/A SAS Address(0): 0x443322110500 Connected Port Number: 1(path0) Inquiry Data: CVPR206200LX300EGN INTEL SSDSA2CW300G3 4PC10362 FDE Capable: Not Capable FDE Enable: Disable Secured: Unsecured Locked: Unlocked Needs EKM Attention: No Foreign State: None Device Speed: 3.0Gb/s Link Speed: 3.0Gb/s Media Type: Solid State Device Drive: Not Certified Drive Temperature : N/A PI Eligibility: No Drive is formatted for PI information: No PI: No PI Drive's NCQ setting : N/A Port-0 : Port status: Active Port's Linkspeed: 3.0Gb/s Drive has flagged a S.M.A.R.T alert : No Exit Code: 0x00 These appear to be in a single raid 1 at this time. We cannot really add a single disk in, without breaking the entire redundancy aspect of the raid/data arrays. So I would recommend that we upgrade by adding two more disks to each from our on site spares, of which we have 38 Intel 320 Series SSDSA2CW300G3 2.5" 300GB. These are the identical model. I suggest we add in two more per system, and then reinstall the system from a single raid1 array into a 4 disk raid10 array. We could do this upgrade to one system and have it come fully back online before installing in the second. The only costs I see is the rolling maintainance/reinstallation of each system (leaving one of the two in full service at all times). We have 38 Intel 320 SSDs, as they are an older model spare we do not use in our newer SSD based deployments. @Smalyshev: Would you please review my above suggestion and provide feedback? If you agree, we can assign this task to @Mark to approve the use of allocated spares for upgrades. (The use of spares for repair is automatic, but stealing spares for upgrade requires we ensure its acceptable.) If you do not agree, please assign this back to me with your corrections. Thanks! TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH added a comment. It would require a reimage to replace the raid1 with a raid10. However, it would be the best option in the long run, as a raid10 can then have additional disk pairs added without reimage. (So adding any further disks in the future shouldn't require reimage, as long as our reinstall this time includes the raid10 and then an LVM before our parititons.) So we would need to coordinate when it could be done, but if that is acceptable to you we can kick this up to @mark for his approval of implementation. TASK DETAIL https://phabricator.wikimedia.org/T119579 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T119579: Additional diskspace of wdqs1001/wdqs1002
RobH moved this task to In Discussion / Review on the hardware-requests workboard. TASK DETAIL https://phabricator.wikimedia.org/T119579 WORKBOARD https://phabricator.wikimedia.org/project/board/1014/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, RobH Cc: RobH, mark, fgiunchedi, hoo, Aklapper, Joe, StudiesWorld, Smalyshev, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T112892: [Bug] analytics stat1002 xmldatadumps wikidata entities symlink incorrect
RobH closed this task as "Resolved" by committing rOPUPfb35e11d8ce4: Make link in dataset relative. TASK DETAIL https://phabricator.wikimedia.org/T112892 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki, RobH Cc: gerritbot, Addshore, Aklapper, Wikidata-bugs, aude, Svick, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T86561: Wikidata Query Service hardware
RobH moved this task to Allocation/Ordering on the hardware-requests workboard. TASK DETAIL https://phabricator.wikimedia.org/T86561 WORKBOARD https://phabricator.wikimedia.org/project/board/1014/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RobH Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude, fgiunchedi, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T86561: Wikidata Query Service hardware
RobH moved this task to In Discussion on the hardware-requests workboard. TASK DETAIL https://phabricator.wikimedia.org/T86561 WORKBOARD https://phabricator.wikimedia.org/project/board/1014/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RobH Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude, fgiunchedi, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T86561: Wikidata Query Service hardware
RobH added a project: hardware-requests. RobH added a subscriber: RobH. TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RobH Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude, fgiunchedi, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T86561: Wikidata Query Service hardware
RobH added a comment. Update from IRC discussion: We'll be allocating two systems for this: Dell PowerEdge https://phabricator.wikimedia.org/diffusion/EPBO/, Dual Intel Xeon https://phabricator.wikimedia.org/E5-2440, 64GB Memory, Dual 300GB SSD, H310 Mini Raid Card wmf3543 - currently in d3-eqiad, and staying there wmf3544 - currently in d3-eqiad, and relocating into another row As such, this task will have a blocker to completion, being the relocation of the second host and its network setup, etc... TASK DETAIL https://phabricator.wikimedia.org/T86561 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RobH Cc: RobH, Matanya, faidon, mark, JanZerebecki, Aklapper, Joe, Smalyshev, Manybubbles, jkroll, Wikidata-bugs, Jdouglas, aude, fgiunchedi, Dzahn, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T105185: Need deploy rights for Wikidata Query Service
RobH moved this task to In Review on the Ops-Access-Requests workboard. TASK DETAIL https://phabricator.wikimedia.org/T105185 WORKBOARD https://phabricator.wikimedia.org/project/board/956/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RobH Cc: Dzahn, gerritbot, Matanya, Aklapper, Westonnh, tomasz, Jdouglas, Joe, Smalyshev, jkroll, Wikidata-bugs, RobH, aude, Manybubbles, mark, JanZerebecki, faidon, fgiunchedi, chasemp, Malyacko, Krenair, P.Copp ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T84902: deploy haedus and capella with os for orientdb testing
RobH added a comment. It seems I accidentally didnt notice and break the raid config setup on capella, as it uses an H310. A quick look confirms haedus does as well. So now capella is reinstalled with the h310 in bypass mode, presenting the disks normally. So this should be good to go, going to remove the codfw onsite flag, as there isn't any onsite work to be done. If these aren't good for this testing, joe mentioned the old lsearch systems would also work. So this is allocated for the testing, but can be swapped if it turns out to not be ideal. TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: Aklapper, RobH, Joe, aaron, jkroll, Smalyshev, Wikidata-bugs, aude, GWicke, Manybubbles, daniel, mark, JohnLewis, faidon, akosiaris, fgiunchedi, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T84902: deploy haedus and capella with os for orientdb testing
RobH removed a project: ops-codfw. TASK DETAIL https://phabricator.wikimedia.org/T84902 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign username. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobH Cc: Aklapper, RobH, Joe, aaron, jkroll, Smalyshev, Wikidata-bugs, aude, GWicke, Manybubbles, daniel, JohnLewis, faidon, akosiaris, fgiunchedi, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
