[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
bd808 added a comment. In T206636#4946372, @Smalyshev wrote: Not sure if virtualization that we do now allows such kind of fixed resource allocations (probably also I/O resources need to be taken care of?) We have a few special instances that we do this for today. It is something that we are experimenting with for virtualization of shared Cloud Services systems such as the ToolsDB databases. The process that requires cloud-root assistance to create the initial instances using cli magic, but it is possible. Basically we mark the cloudvirt as unavailable for selection by the normal OpenStack scheduler and then force create the desired instances there manually.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: bd808Cc: bd808, Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Yes, judging from our preliminary test, if we get uncontested use of the server or even a certain chunk of it maybe (not sure if possible?) it would be enough. Note that an interesting scenario that we want to test in foreseeable future involves cluster setup, so we'd want at least 2 hosts (not sure whether they have to be on 2 separate hardware machines) with requirements close to what wdqs hosts have. It could splitting cloudvirt host into two VMs exclusively used by these test hosts would be ok. Not sure if virtualization that we do now allows such kind of fixed resource allocations (probably also I/O resources need to be taken care of?)TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: bd808, Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
bd808 added a comment.
The next step here is to determine the hardware requirements and then find out if there is current fiscal year budget to cover procuring the required hardware, or if we can allocate a dedicated server from the existing cloudvirt pool and backfill that capacity.
The wdqs10{09,10} prod hosts each have 8 cores, 128GB RAM, 4x800GB SSD (1.6TB RAID10). Our most recent cloudvirt servers (cloudvirt10[25-30]) are 36 physical cores, 512GB RAM, 6x1.92TB SSD (5.7TB RAID10). I think this means that a single cloudvirt would be more than enough hardware if it was not shared with other arbitrary workloads.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: bd808Cc: bd808, Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi, chasemp___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. In the meantime, can I delete t206636-3? Yes.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. So it sounds like you will need dedicated hardware to make this work (if the original test with the t206636 VM was adequate.) The analytics team is experimenting with this model currently so we'll see how it goes. You would need to figure out about budgeting/procurement/specs/etc to get your own server installed within the cloud but that's not especially complicated. In the meantime, can I delete t206636-3?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Tried with incoming stream, the machine can't keep up - by now it's 5 hours behind, and processing about half the necessary requests. I think the conclusion is mostly clear - this VM as such is not suitable for any performance testing or any workloads that are close to what we have in production, and can only be used for feature testing. @Andrew is there any way to improve the performance?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Updater seems to be able to get about 4-5 updates per second, which is abut 2x slower than production. Summarily, it looks like this setup may be fit for functionality testing, but decidedly unfit for any performance testing, as it is 2-3x slower than production. Final test I want to do it to see if the machine is capable of dealing with incoming update stream (without having to catch up). After that, it can be shut down.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Loading finished, overall took 8 days and 9 hours, or 201 hours, or 3x compared to production. Launching updater next to see the updates speed.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Looks like this VM is substantially slower - I started data load at Nov 9, now is end of Nov 13, and it's only 75% done, which means it'll take about a week to load all data, which is significantly slower than in production (it was done in 63 hours). However, the real test would be the update/query speed, so I'll wait until the loading is finished and see how the Updater is doing and how querying works.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Legado_Shulgin, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, D3r1ck01, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Data loading test launched for t206636-3 and should take several days to complete. Ideally, it should not be interrupted. Please ping me if VM needs to be rebooted, etc. - it's possible to recover but to have accurate record of timings I need to know when it happened.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. I've build a new VM t206636-3 that should have the same specs as your original test VM. Is it OK if I delete -2 now? Yes, sure. I'll use t206636-3.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. In T206636#4690379, @Smalyshev wrote: I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs I see that t206636-2 is listed as having 4 VPU and 24G RAM. That sounds too small for what I'd need - is this accurate? That's my mistake -- I've build a new VM t206636-3 that should have the same specs as your original test VM. Is it OK if I delete -2 now?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Gehel added a comment. In T206636#4690384, @Smalyshev wrote: @Andrew Also looks like there is some puppet issue there: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item lvs::configuration::lvs_service_ips in any Hiera data file and no default supplied at /etc/puppet/modules/lvs/manifests/configuration.pp:71:20 on node t206636-2.wikidata-query.eqiad.wmflabs Judging from the names of the settings, it's not one of mine, so could you please look into what is going on there? Looks like you are using role::wdqs which has some production specific things (like LVS configuration). For labs, we have role::wdqs::labs which should work fine.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. @Andrew Also looks like there is some puppet issue there: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item lvs::configuration::lvs_service_ips in any Hiera data file and no default supplied at /etc/puppet/modules/lvs/manifests/configuration.pp:71:20 on node t206636-2.wikidata-query.eqiad.wmflabs Judging from the names of the settings, it's not one of mine, so could you please look into what is going on there?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs I see that t206636-2 is listed as having 4 VPU and 24G RAM. That sounds too small for what I'd need - is this accurate?TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. I've created a new VM, t206636-2.wikidata-query.eqiad.wmflabs. This is in the older region, on a host that is not super busy but is supporting quite a few other VMs. If your tests look good there too then we're probably in good shape and can avoid needing special hardware just for you.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. Run a second set of tests on a similar VM that shares a host with other active cloud instances (to see if we can meet your needs as a standard user), or Sure. This would take a bit more time though - we'll need to load a new dump (probably several days) and run updater for a few days and queries on top of it. So overall I estimate it will take 1.5-2 weeks clock time (though mostly it'd be waiting for stuff to complete). I am interested in such test, so let's try it.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. Thanks, Stas. There are two ways I think we can go forward with this: Run a second set of tests on a similar VM that shares a host with other active cloud instances (to see if we can meet your needs as a standard user), or Start talking about buying you a private virt host so you can do tests like this that are uncontested by anything except for other tests of yours Number 1 seems like a good option to eliminate but of course it means you'll need to repeat your tests. Unfortunately to get a good test in the new future I'd need to build you a new VM in the old region (where there's actually some busy traffic happening). If you're interested in giving that a shot, let me know; otherwise we'll need to pass this issue on to some budget-focused people to discuss #2.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. I did a short evaluation on provided VM and it looks like it behaves well enough to run Blazegraph tests on it. So if we get something like that, we could probably do tests there. The t206636 can be shut down now, if necessary.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. note to self, I can merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/468377/ after Stas releases this VM (or at least stops caring about resource contention)TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. I definitely see role::wdqs::labs as an available class in Horizon, in the 'Puppet Configuration' tab. I found it, I was looking at the wrong region. Dumps are ok now too, so I think I've got enough to start testing.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. @Smalyshev, dumps should be available on that host now.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Andrew added a comment. Sorry! Since you mentioned having tested this on a VM in the past I assumed you had a plan in mind to set something up for testing. All of the storage for that instance is hosted on SSDs. I can partition the extra space for you if you want -- you would just want to add the puppet class profile::labs::lvm::srv. I definitely see role::wdqs::labs as an available class in Horizon, in the 'Puppet Configuration' tab. I'll have a look at what's happening with the dumps mount.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AndrewCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. @Andrew I logged in there and I see this set of disks: Filesystem Size Used Avail Use% Mounted on udev65G 0 65G 0% /dev tmpfs 13G 19M 13G 1% /run /dev/vda3 19G 1.5G 17G 9% / tmpfs 65G 0 65G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 65G 0 65G 0% /sys/fs/cgroup labstore1003.eqiad.wmnet:/scratch 3.0T 1.5T 1.4T 51% /mnt/nfs/labstore1003-scratch labstore1006.wikimedia.org:/dumps 916G 2.6G 867G 1% /mnt/nfs/dumps-labstore1006.wikimedia.org labstore1007.wikimedia.org:/dumps 916G 203G 667G 24% /mnt/nfs/dumps-labstore1007.wikimedia.org tmpfs 13G 0 13G 0% /run/user/10977 I am not sure which one is the SSD space? Is it connected? Also, it looks like /mnt/nfs/dumps-labstore1007.wikimedia.org is empty. It'd be nice to get access to public dumps there, like on wdqs-test.eqiad.wmflabs.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T206636: Provide a way to have test servers on real hardware, isolated from production for Wikidata Query Service
Smalyshev added a comment. @Andrew thanks, I'll start testing on it tomorrow.TASK DETAILhttps://phabricator.wikimedia.org/T206636EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Addshore, Andrew, Aklapper, Smalyshev, Gehel, Nandana, thifranc, AndyTan, sietec, Davinaclare77, Qtn1293, Lahi, aborrero, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Jonas, Xmlizer, Wong128hk, Luke081515, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, bd808, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
