On Fri, Jul 8, 2016 at 3:24 PM, Chase Pettet <[email protected]> wrote: > In case some of of the readers of this list are unaware, new instance > creation is currently suspended due to capacity. We have some new resources > we are working diligently to use for expanding but they are still a limited > solution.
The fact that the servers we have operational now are full is actually great because it means that people are using Labs more than ever before. It's not great that the timing of several things conspired to keep additional resources from being deployed before we hit a critical point of usage. I think that historically the collective "we" have done a poor job of recognizing the cost and savings associated with use of the Labs environment. On the cost side, thus far budget for hardware in Labs has been virtually unlimited. By that I mean that as far as I know any time more hardware has been asked for the budget has been found to supply it. Human resources however are not so plentiful. Full time staffing has never been higher than 3 engineers and often only 2 in practice. This is just a reality of the WMF budget process. On the savings side, some really really rough napkin math using calculator.s3.amazonaws.com shows that just the virtual machines in active use by Labs projects would cost between $45k and $50k USD per month. On top of that a real AWS user would have to pay for data transfer and other networking services and probably a few support hours. That $50k USD per month also doesn't include any cost for things like actively managed real-time replicas of the Wikimedia production databases or any of the other shared services available in Labs. All told I would say that the Wikimedia movement is getting more than fair return on the moneys invested in the Labs hardware and staffing. I personally think that line items for Labs and Tool Labs are some of the best spent dollars in the Wikimedia Foundation budget due to the incredibly diverse and productive projects that they enable volunteers to create. But I get a bit crabby when people act as though AGF only applies on wiki. When Chase, Andrew, Yuvi, or anyone else spending their time and energy to keep Labs running asks for a change of the status quo they are asking with good intent. When you spin up a new VM or launch a grid job in Tool Labs there is an opportunity cost to the Wikimedia movement that should be respected. Every compute hour or gigabyte of storage your personal project consumes is one that is not available for someone else. Shutting down unused systems and cleaning up projects that have run their course is a reasonable thing to ask. Bryan -- Bryan Davis Wikimedia Foundation <[email protected]> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
