[Wikidata-bugs] [Maniphest] [Updated] T169133: WDQS testing setup platform sizing
Andrew added a project: Cloud-VPS (Quota-requests). TASK DETAILhttps://phabricator.wikimedia.org/T169133EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: daniel, Lydia_Pintscher, bd808, chasemp, Gehel, Aklapper, Smalyshev, GoranSMilovanovic, QZanden, EBjune, Tbscho, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, scfc, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T169133: WDQS testing setup platform sizing
Smalyshev added a comment. I have not yet heard an explanation of the resources needed Sure, sorry for not specifying it early. So, in production we have: Database size now is 193G, 75% growth per year for now, though not sure it will keep growing this way for years. We have 500 to 700G of diskspace on production hosts, so I'd say we need about 500G on test host too (though we could reduce the data set on test DB, possibly, but the growth will still remain). Production uses (and requires) SSD, but I think for test host spinning disk would be ok. Memory - directly related to performance, so the more the better, but it can run inside 16G, though something like 64G would probably work better. CPU - here's it a bit hard for me to say. The workload is mostly I/O (disk) bound, production has Dual Xeon E5-2620 v3 @2.4Ghz, but it's hard to see what we need in labs since we won't be serving any traffic. What we have in production is probably overkill for test setup, but what we have now in labs with 8 cores can't reliably keep up with update steam (maybe not CPU but I/O issue). Summarily, what is needed is: Much more diskspace and I/O throughput Somewhat more memory Maybe more CPU power, not sure - need to see TASK DETAILhttps://phabricator.wikimedia.org/T169133EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: daniel, Lydia_Pintscher, bd808, chasemp, Gehel, Aklapper, Smalyshev, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T169133: WDQS testing setup platform sizing
bd808 added a comment. Create a huge storage labs instance (300G, 500G) template It is likely that #labs can find space to grant a quota in the 300-500G range. To do that we need to have a few specifics about the exact sizing desired so we can look for a labvirt host that can support the local storage needs and provide the IOPS needed as well. The normal process for this is to file a subtask of T140904: Existing Labs project quota increase requests (Tracking). The request really does need to explain what the additional resources will be used for so that the process of allocating a larger than normal component of our shared pool of resources is documented and transparent. Migrate test setup to real hardware outside of labs (maybe less high-powered than production hosts) - maybe to labs-support like relforge I was not a part of the negotiations that led to the creation of Relforge, but it is a very unique situation within the Cloud Services projects. A very detailed case will need to be made for us to consider adding another special service like this. I would really be more inclined to look at adding dedicated labvirt nodes before I gave the ok to placing more custom boxes into the labs-support vlan.TASK DETAILhttps://phabricator.wikimedia.org/T169133EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: bd808Cc: daniel, Lydia_Pintscher, bd808, chasemp, Gehel, Aklapper, Smalyshev, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
