Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Fri, Feb 28, 2020 at 4:38 AM Dave Airlie wrote: > > On Fri, 28 Feb 2020 at 07:27, Daniel Vetter wrote: > > > > Hi all, > > > > You might have read the short take in the X.org board meeting minutes > > already, here's the long version. > > > > The good news: gitlab.fd.o has become very popular with our > > communities, and is used extensively. This especially includes all the > > CI integration. Modern development process and tooling, yay! > > > > The bad news: The cost in growth has also been tremendous, and it's > > breaking our bank account. With reasonable estimates for continued > > growth we're expecting hosting expenses totalling 75k USD this year, > > and 90k USD next year. With the current sponsors we've set up we can't > > sustain that. We estimate that hosting expenses for gitlab.fd.o > > without any of the CI features enabled would total 30k USD, which is > > within X.org's ability to support through various sponsorships, mostly > > through XDC. > > > > Note that X.org does no longer sponsor any CI runners themselves, > > we've stopped that. The huge additional expenses are all just in > > storing and serving build artifacts and images to outside CI runners > > sponsored by various companies. A related topic is that with the > > growth in fd.o it's becoming infeasible to maintain it all on > > volunteer admin time. X.org is therefore also looking for admin > > sponsorship, at least medium term. > > > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > > (without CI support) and a trimmed XDC and assuming no sponsor payment > > meanwhile, we'd have to cut CI services somewhere between May and June > > this year. The board is of course working on acquiring sponsors, but > > filling a shortfall of this magnitude is neither easy nor quick work, > > and we therefore decided to give an early warning as soon as possible. > > Any help in finding sponsors for fd.o is very much appreciated. > > a) Ouch. > > b) we probably need to take a large step back here. > > Look at this from a sponsor POV, why would I give X.org/fd.o > sponsorship money that they are just giving straight to google to pay > for hosting credits? Google are profiting in some minor way from these > hosting credits being bought by us, and I assume we aren't getting any > sort of discounts here. Having google sponsor the credits costs google > substantially less than having any other company give us money to do > it. > > If our current CI architecture is going to burn this amount of money a > year and we hadn't worked this out in advance of deploying it then I > suggest the system should be taken offline until we work out what a > sustainable system would look like within the budget we have, whether > that be never transferring containers and build artifacts from the > google network, just having local runner/build combos etc. Google has sponsored 30k in hosting credits last year, these simply ran out _much_ faster than anyone planned for. So this is by far not a free thing for them. Plus there's also other companies sponsoring CI runners and what not else in equally substantial amounts, plus the biggest thing, sponsored admin time (more or less officially). So there's a _lot_ of room for companies like Red Hat to sponsor without throwing any money in google's revenue stream. Or it doesn't happen, and then yeah the decision has already been made to shutter the CI services. So this is also a question of whether we (as a community and all the companies benefitting from the work done) really want this, or maybe not quite. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] [Intel-gfx] gitlab.fd.o financial situation and impact on services
On Fri, 28 Feb 2020 at 07:27, Daniel Vetter wrote: > > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. a) Ouch. b) we probably need to take a large step back here. Look at this from a sponsor POV, why would I give X.org/fd.o sponsorship money that they are just giving straight to google to pay for hosting credits? Google are profiting in some minor way from these hosting credits being bought by us, and I assume we aren't getting any sort of discounts here. Having google sponsor the credits costs google substantially less than having any other company give us money to do it. If our current CI architecture is going to burn this amount of money a year and we hadn't worked this out in advance of deploying it then I suggest the system should be taken offline until we work out what a sustainable system would look like within the budget we have, whether that be never transferring containers and build artifacts from the google network, just having local runner/build combos etc. Dave. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On 02/27/2020 05:00 PM, Tom Stellard wrote: > On 02/27/2020 01:27 PM, Daniel Vetter wrote: >> Hi all, >> >> You might have read the short take in the X.org board meeting minutes >> already, here's the long version. >> >> The good news: gitlab.fd.o has become very popular with our >> communities, and is used extensively. This especially includes all the >> CI integration. Modern development process and tooling, yay! >> >> The bad news: The cost in growth has also been tremendous, and it's >> breaking our bank account. With reasonable estimates for continued >> growth we're expecting hosting expenses totalling 75k USD this year, >> and 90k USD next year. With the current sponsors we've set up we can't >> sustain that. We estimate that hosting expenses for gitlab.fd.o >> without any of the CI features enabled would total 30k USD, which is >> within X.org's ability to support through various sponsorships, mostly >> through XDC. >> > > Have you looked into applying for free credits from amazon: > > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ > Also fastly provides free CDN services to some Open Source projects: https://www.fastly.com/open-source?utm_medium=social_source=t.co_campaign=FY17Q4_WebPage_OpenSource It might also be worth looking into if the main costs are coming from data transfers. -Tom > -Tom > >> Note that X.org does no longer sponsor any CI runners themselves, >> we've stopped that. The huge additional expenses are all just in >> storing and serving build artifacts and images to outside CI runners >> sponsored by various companies. A related topic is that with the >> growth in fd.o it's becoming infeasible to maintain it all on >> volunteer admin time. X.org is therefore also looking for admin >> sponsorship, at least medium term. >> >> Assuming that we want cash flow reserves for one year of gitlab.fd.o >> (without CI support) and a trimmed XDC and assuming no sponsor payment >> meanwhile, we'd have to cut CI services somewhere between May and June >> this year. The board is of course working on acquiring sponsors, but >> filling a shortfall of this magnitude is neither easy nor quick work, >> and we therefore decided to give an early warning as soon as possible. >> Any help in finding sponsors for fd.o is very much appreciated. >> >> Thanks, Daniel >> > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On 02/27/2020 01:27 PM, Daniel Vetter wrote: > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > Have you looked into applying for free credits from amazon: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ -Tom > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, 27 Feb 2020 22:27:04 +0100 Daniel Vetter said: Might I suggest that given the kind of expenses detailed here, literally buying 1 - 4 reasonably specced boxes and hosting them at OSUOSL would be incredibly cheaper? (we (enlightenment.org) have been doing so for years on a single box). We farm out CI to travis via gihub mirrors as it's not considered an essential core service (unlike mailing lists, git, phabricator whch nwe still run - we can live without CI for a while and find other ways). The cost is the odd HDD replacement every few years and maybe every 10y or so a new box. That's a massively lower cost than you are quoting below. OSUOSL provide bandwidth, power, rack space etc. for free. They have been fantastic IMHO and the whole "no fat bills" is awesome and you get a full system to set up any way you like. You just bring the box. That should drop cost through the floor. It will require some setup and admin though. > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel > -- > Daniel Vetter > Software Engineer, Intel Corporation > +41 (0) 79 365 57 48 - http://blog.ffwll.ch > ___ > xorg-de...@lists.x.org: X.Org development > Archives: http://lists.x.org/archives/xorg-devel > Info: https://lists.x.org/mailman/listinfo/xorg-devel > -- - Codito, ergo sum - "I code, therefore I am" -- Carsten Haitzler - ras...@rasterman.com ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, Feb 27, 2020 at 10:27:04PM +0100, Daniel Vetter wrote: > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel So this cost is all about fd.o? I feel that this was not communicated to x.org members at all, when fd.o "merging" was suggested. In as far as this was a merge. Luc Verhaegen. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, Feb 27, 2020 at 1:27 PM Daniel Vetter wrote: > > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. Some clarification I got from Daniel in a private conversation, since I was confused about what the money was paying for exactly: We're paying 75K USD for the bandwidth to transfer data from the GitLab cloud instance. i.e., for viewing the https site, for cloning/updating git repos, and for downloading CI artifacts/images to the testing machines (AFAIU). I was not aware that we were being charged for anything wrt GitLab hosting yet (and neither was anyone on my team at Intel that I've asked). This... kind of needs to be communicated. A consistent concern put forth when we were discussing switching to GitLab and building CI was... how do we pay for it. It felt like that concern was always handwaved away. I heard many times that if we needed more runners that we could just ask Google to spin up a few more. If we needed testing machines they'd be donated. No one mentioned that all the while we were paying for bandwidth... Perhaps people building the CI would make different decisions about its structure if they knew it was going to wipe out the bank account. What percentage of the bandwidth is consumed by transferring CI images, etc? Wouldn't 75K USD would be enough to buy all the testing machines we need and host them within Google or wherever so we don't need to pay for huge amounts of bandwidth? I understand that self-hosting was attractive so that we didn't find ourselves on the SourceForge-equivalent hosting platform of 2022, but is that risk real enough to justify spending 75K+ per year? If we were hosted on gitlab.com or github.com, we wouldn't be paying for transferring CI images to CI test machines, etc, would we? So what do we do now? Have we painted ourselves into a corner? ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] gitlab.fd.o financial situation and impact on services
Hi all, You might have read the short take in the X.org board meeting minutes already, here's the long version. The good news: gitlab.fd.o has become very popular with our communities, and is used extensively. This especially includes all the CI integration. Modern development process and tooling, yay! The bad news: The cost in growth has also been tremendous, and it's breaking our bank account. With reasonable estimates for continued growth we're expecting hosting expenses totalling 75k USD this year, and 90k USD next year. With the current sponsors we've set up we can't sustain that. We estimate that hosting expenses for gitlab.fd.o without any of the CI features enabled would total 30k USD, which is within X.org's ability to support through various sponsorships, mostly through XDC. Note that X.org does no longer sponsor any CI runners themselves, we've stopped that. The huge additional expenses are all just in storing and serving build artifacts and images to outside CI runners sponsored by various companies. A related topic is that with the growth in fd.o it's becoming infeasible to maintain it all on volunteer admin time. X.org is therefore also looking for admin sponsorship, at least medium term. Assuming that we want cash flow reserves for one year of gitlab.fd.o (without CI support) and a trimmed XDC and assuming no sponsor payment meanwhile, we'd have to cut CI services somewhere between May and June this year. The board is of course working on acquiring sponsors, but filling a shortfall of this magnitude is neither easy nor quick work, and we therefore decided to give an early warning as soon as possible. Any help in finding sponsors for fd.o is very much appreciated. Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
