[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

EBernhardson added a comment. Thanks! TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Andrew, EBernhardson Cc: EBernhardson, JHedden, Iamamz3, GreenReaper, RazShuty, Krenair, Smalyshev,

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Andrew added a comment. @EBernhardson sorry for the delay in responding! We can return these hosts to you at any time -- I'll start working on that shortly unless you're back to not needing them :) TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

EBernhardson added a comment. Priorities have changed again, commons query service is back at the top of the stack. How are we in terms of a timeline making these available? Have we finished up the ceph testing? TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2020-03-04T20:03:31Z] add cloudvirt-wdqs100[123] to ceph host aggregate T221631 TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

gerritbot added a comment. Change 576903 **merged** by Jhedden: [operations/puppet@production] openstack: switch cloudvirt-wdqs servers to Ceph https://gerrit.wikimedia.org/r/576903 TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

JHedden added a comment. These hosts will temporarily be used for testing CloudVPS's new Ceph storage cluster. To switch back to local storage, we'll need to migrate or delete all running virtual machines and revert this patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/576903

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

gerritbot added a comment. Change 576903 had a related patch set uploaded (by Jhedden; owner: Jhedden): [operations/puppet@production] openstack: switch cloudvirt-wdqs servers to Ceph https://gerrit.wikimedia.org/r/576903 TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Andrew added a comment. breadcrumbs for whoever picks this up in June: I created a 'wdqs-scaling ' project and a custom flavor 'wdqs-scaling' which is associated with the 'wdqs' host aggregate and is just big enough to fill up one of these servers with a single giant VM. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Andrew added a comment. @Gehel after a long time in dc-limbo these hosts are just about ready to go. I'm in the process of setting up a project and custom VM flavor for you to use., Do you have a sense of when you'll have time to pay attention to the new systems? If at this point you

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['cloudvirt-wdqs1002.eqiad.wmnet'] and were **ALL** successful. TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['cloudvirt-wdqs1003.eqiad.wmnet'] and were **ALL** successful. TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: cloudvirt-wdqs1002.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202002272018_andrew_134870_cloudvirt-wdqs1002_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: cloudvirt-wdqs1003.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202002272013_andrew_133073_cloudvirt-wdqs1003_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['cloudvirt-wdqs1003.eqiad.wmnet'] Of which those **FAILED**: ['cloudvirt-wdqs1003.eqiad.wmnet'] TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['cloudvirt-wdqs1002.eqiad.wmnet'] Of which those **FAILED**: ['cloudvirt-wdqs1002.eqiad.wmnet'] TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Completed auto-reimage of hosts: ['cloudvirt-wdqs1002.eqiad.wmnet'] Of which those **FAILED**: ['cloudvirt-wdqs1002.eqiad.wmnet'] TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: cloudvirt-wdqs1003.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202002271950_andrew_128639_cloudvirt-wdqs1003_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: cloudvirt-wdqs1002.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202002271950_andrew_128610_cloudvirt-wdqs1002_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

ops-monitoring-bot added a comment. Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts: cloudvirt-wdqs1002.eqiad.wmnet The log can be found in `/var/log/wmf-auto-reimage/202002271949_andrew_128500_cloudvirt-wdqs1002_eqiad_wmnet.log`. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

gerritbot added a comment. Change 575312 **merged** by Andrew Bogott: [operations/puppet@production] Add cloudvirt-wdqs hosts https://gerrit.wikimedia.org/r/575312 TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

gerritbot added a comment. Change 575312 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott): [operations/puppet@production] Add cloudvirt-wdqs hosts https://gerrit.wikimedia.org/r/575312 TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Smalyshev added a comment. Both evaluating Virtuoso and other solutions (like JanusGraph) would require that. @Gehel should know the details. TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

bd808 added a comment. I verified today that we have budget for this build in FY19/20 Q1 (July-September 2019). Before I start a procurement request, @Gehel can you confirm that this test cluster is still desired by the Search team? TASK DETAIL https://phabricator.wikimedia.org/T221631

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

bd808 added a comment. In T221631#5131369 , @Gehel wrote: > So my preference would be to have dedicated servers, with a single VM on each, basically using our cloud infrastructure only as an isolation layer. I'm not sure how much sense

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Gehel added a comment. Yes, this is the continuation of T206636 . In T221631#5131322 , @bd808 wrote: > 1. Try to map these requirements into our "standard" cloudvirt hosts and expand our

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

bd808 added a comment. I see 2 ways to approach this: 1. Try to map these requirements into our "standard" cloudvirt hosts and expand our FY2019/2020 planned growth to include the needed expansion 2. Spec and rack specific hardware sized for these instances using the "fake ironic"

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

bd808 added a comment. In T206636#4941969 , @bd808 wrote: > The next step here is to determine the hardware requirements and then find out if there is current fiscal year budget to cover procuring the required hardware, or if we can

[Wikidata-bugs] [Maniphest] [Commented On] T221631: Dedicated servers on WMCS to test WDQS scalability strategy

Andrew added a comment. Can you tell me more about what these hosts would actually do? Are they virt hosts hosting one big VM each? TASK DETAIL https://phabricator.wikimedia.org/T221631 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Andrew Cc: