Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Mon, Apr 6, 2020 at 10:04 AM Michel Dänzer wrote: > > On 2020-04-06 6:34 p.m., Rob Clark wrote: > > > > The ideal thing would be to be able to click any jobs that we want to > > run, say "arm64_a630_gles31", and for gitlab to realize that it needs > > to automatically trigger dependencies of that job (meson-arm64, and > > arm_build+arm_test). But not sure if that is a thing gitlab can do. > > Not that I know of. The dependency handling is still pretty rudimentary > in general. > > > > Triggering just the first container jobs and having everything from > > there run automatically would be ok if the current rules to filter out > > unneeded jobs still apply, ie. a panfrost change isn't triggering > > freedreno CI jobs and visa versa. But tbh I don't understand enough > > of what that MR is doing to understand if that is what it does. (It > > was suggested on IRC that this is probably what it does.) > > It is. Filtered jobs don't exist at all in the pipeline, so they can't > be triggered by any means. :) > ahh, ok, I didn't realize that.. thx for the explaination BR, -R ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On 2020-04-06 6:34 p.m., Rob Clark wrote: > > The ideal thing would be to be able to click any jobs that we want to > run, say "arm64_a630_gles31", and for gitlab to realize that it needs > to automatically trigger dependencies of that job (meson-arm64, and > arm_build+arm_test). But not sure if that is a thing gitlab can do. Not that I know of. The dependency handling is still pretty rudimentary in general. > Triggering just the first container jobs and having everything from > there run automatically would be ok if the current rules to filter out > unneeded jobs still apply, ie. a panfrost change isn't triggering > freedreno CI jobs and visa versa. But tbh I don't understand enough > of what that MR is doing to understand if that is what it does. (It > was suggested on IRC that this is probably what it does.) It is. Filtered jobs don't exist at all in the pipeline, so they can't be triggered by any means. :) -- Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Friday 2020-02-28 08:59, Daniel Stone wrote: > >I believe that in January, we had $2082 of network cost (almost >entirely egress; ingress is basically free) and $1750 of >cloud-storage cost (almost all of which was download). That's based >on 16TB of cloud-storage (CI artifacts, container images, file >uploads, Git LFS) egress and 17.9TB of other egress (the web service >itself, repo activity). Projecting that out [×12 for a year] gives >us roughly $45k of network activity alone, I had come to a similar conclusion a few years back: It is not very economic to run ephemereal buildroots (and anything like it) between two (or more) "significant locations" of which one end is located in a Large Cloud datacenter like EC2/AWS/etc. As for such usecases, me and my surrounding peers have used (other) offerings where there is 50 TB free network/month, and yes that may have entailed doing more adminning than elsewhere - but an admin appreciates $2000 a lot more than a corporation, too. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Fri, Feb 28, 2020 at 12:00 AM Daniel Stone wrote: > > Hi Matt, > > On Thu, 27 Feb 2020 at 23:45, Matt Turner wrote: > > We're paying 75K USD for the bandwidth to transfer data from the > > GitLab cloud instance. i.e., for viewing the https site, for > > cloning/updating git repos, and for downloading CI artifacts/images to > > the testing machines (AFAIU). > > I believe that in January, we had $2082 of network cost (almost > entirely egress; ingress is basically free) and $1750 of cloud-storage > cost (almost all of which was download). That's based on 16TB of > cloud-storage (CI artifacts, container images, file uploads, Git LFS) > egress and 17.9TB of other egress (the web service itself, repo > activity). Projecting that out gives us roughly $45k of network > activity alone, so it looks like this figure is based on a projected > increase of ~50%. > > The actual compute capacity is closer to $1150/month. Could we have the full GCP bill posted? ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
Hi Jan, On Fri, 28 Feb 2020 at 10:09, Jan Engelhardt wrote: > On Friday 2020-02-28 08:59, Daniel Stone wrote: > >I believe that in January, we had $2082 of network cost (almost > >entirely egress; ingress is basically free) and $1750 of > >cloud-storage cost (almost all of which was download). That's based > >on 16TB of cloud-storage (CI artifacts, container images, file > >uploads, Git LFS) egress and 17.9TB of other egress (the web service > >itself, repo activity). Projecting that out [×12 for a year] gives > >us roughly $45k of network activity alone, > > I had come to a similar conclusion a few years back: It is not very > economic to run ephemereal buildroots (and anything like it) between > two (or more) "significant locations" of which one end is located in > a Large Cloud datacenter like EC2/AWS/etc. > > As for such usecases, me and my surrounding peers have used (other) > offerings where there is 50 TB free network/month, and yes that may > have entailed doing more adminning than elsewhere - but an admin > appreciates $2000 a lot more than a corporation, too. Yes, absolutely. For context, our storage & network costs have increased >10x in the past 12 months (~$320 Jan 2019), >3x in the past 6 months (~$1350 July 2019), and ~2x in the past 3 months (~$2000 Oct 2019). I do now (personally) think that it's crossed the point at which it would be worthwhile paying an admin to solve the problems that cloud services currently solve for us - which wasn't true before. Such an admin could also deal with things like our SMTP delivery failure rate, which in the past year has spiked over 50% (see previous email), demand for new services such as Discourse which will enable user support without either a) users having to subscribe to a mailing list, or b) bug trackers being cluttered up with user requests and other non-bugs, etc. Cheers, Daniel ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
Hi Matt, On Thu, 27 Feb 2020 at 23:45, Matt Turner wrote: > We're paying 75K USD for the bandwidth to transfer data from the > GitLab cloud instance. i.e., for viewing the https site, for > cloning/updating git repos, and for downloading CI artifacts/images to > the testing machines (AFAIU). I believe that in January, we had $2082 of network cost (almost entirely egress; ingress is basically free) and $1750 of cloud-storage cost (almost all of which was download). That's based on 16TB of cloud-storage (CI artifacts, container images, file uploads, Git LFS) egress and 17.9TB of other egress (the web service itself, repo activity). Projecting that out gives us roughly $45k of network activity alone, so it looks like this figure is based on a projected increase of ~50%. The actual compute capacity is closer to $1150/month. > I was not aware that we were being charged for anything wrt GitLab > hosting yet (and neither was anyone on my team at Intel that I've > asked). This... kind of needs to be communicated. > > A consistent concern put forth when we were discussing switching to > GitLab and building CI was... how do we pay for it. It felt like that > concern was always handwaved away. I heard many times that if we > needed more runners that we could just ask Google to spin up a few > more. If we needed testing machines they'd be donated. No one > mentioned that all the while we were paying for bandwidth... Perhaps > people building the CI would make different decisions about its > structure if they knew it was going to wipe out the bank account. The original answer is that GitLab themselves offered to sponsor enough credit on Google Cloud to get us started. They used GCP themselves so they could assist us (me) in getting bootstrapped, which was invaluable. After that, Google's open-source program office offered to sponsor us for $30k/year, which was I believe last April. Since then the service usage has increased roughly by a factor of 10, so our 12-month sponsorship is no longer enough to cover 12 months. > What percentage of the bandwidth is consumed by transferring CI > images, etc? Wouldn't 75K USD would be enough to buy all the testing > machines we need and host them within Google or wherever so we don't > need to pay for huge amounts of bandwidth? Unless the Google Cloud Platform starts offering DragonBoards, it wouldn't reduce our bandwidth usage as the corporate network is treated separately for egress. Cheers, Daniel ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On 02/27/2020 05:00 PM, Tom Stellard wrote: > On 02/27/2020 01:27 PM, Daniel Vetter wrote: >> Hi all, >> >> You might have read the short take in the X.org board meeting minutes >> already, here's the long version. >> >> The good news: gitlab.fd.o has become very popular with our >> communities, and is used extensively. This especially includes all the >> CI integration. Modern development process and tooling, yay! >> >> The bad news: The cost in growth has also been tremendous, and it's >> breaking our bank account. With reasonable estimates for continued >> growth we're expecting hosting expenses totalling 75k USD this year, >> and 90k USD next year. With the current sponsors we've set up we can't >> sustain that. We estimate that hosting expenses for gitlab.fd.o >> without any of the CI features enabled would total 30k USD, which is >> within X.org's ability to support through various sponsorships, mostly >> through XDC. >> > > Have you looked into applying for free credits from amazon: > > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ > Also fastly provides free CDN services to some Open Source projects: https://www.fastly.com/open-source?utm_medium=social_source=t.co_campaign=FY17Q4_WebPage_OpenSource It might also be worth looking into if the main costs are coming from data transfers. -Tom > -Tom > >> Note that X.org does no longer sponsor any CI runners themselves, >> we've stopped that. The huge additional expenses are all just in >> storing and serving build artifacts and images to outside CI runners >> sponsored by various companies. A related topic is that with the >> growth in fd.o it's becoming infeasible to maintain it all on >> volunteer admin time. X.org is therefore also looking for admin >> sponsorship, at least medium term. >> >> Assuming that we want cash flow reserves for one year of gitlab.fd.o >> (without CI support) and a trimmed XDC and assuming no sponsor payment >> meanwhile, we'd have to cut CI services somewhere between May and June >> this year. The board is of course working on acquiring sponsors, but >> filling a shortfall of this magnitude is neither easy nor quick work, >> and we therefore decided to give an early warning as soon as possible. >> Any help in finding sponsors for fd.o is very much appreciated. >> >> Thanks, Daniel >> > > ___ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On 02/27/2020 01:27 PM, Daniel Vetter wrote: > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > Have you looked into applying for free credits from amazon: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ -Tom > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel > ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, 27 Feb 2020 22:27:04 +0100 Daniel Vetter said: Might I suggest that given the kind of expenses detailed here, literally buying 1 - 4 reasonably specced boxes and hosting them at OSUOSL would be incredibly cheaper? (we (enlightenment.org) have been doing so for years on a single box). We farm out CI to travis via gihub mirrors as it's not considered an essential core service (unlike mailing lists, git, phabricator whch nwe still run - we can live without CI for a while and find other ways). The cost is the odd HDD replacement every few years and maybe every 10y or so a new box. That's a massively lower cost than you are quoting below. OSUOSL provide bandwidth, power, rack space etc. for free. They have been fantastic IMHO and the whole "no fat bills" is awesome and you get a full system to set up any way you like. You just bring the box. That should drop cost through the floor. It will require some setup and admin though. > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel > -- > Daniel Vetter > Software Engineer, Intel Corporation > +41 (0) 79 365 57 48 - http://blog.ffwll.ch > ___ > xorg-de...@lists.x.org: X.Org development > Archives: http://lists.x.org/archives/xorg-devel > Info: https://lists.x.org/mailman/listinfo/xorg-devel > -- - Codito, ergo sum - "I code, therefore I am" -- Carsten Haitzler - ras...@rasterman.com ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, Feb 27, 2020 at 10:27:04PM +0100, Daniel Vetter wrote: > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. > > Thanks, Daniel So this cost is all about fd.o? I feel that this was not communicated to x.org members at all, when fd.o "merging" was suggested. In as far as this was a merge. Luc Verhaegen. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] gitlab.fd.o financial situation and impact on services
On Thu, Feb 27, 2020 at 1:27 PM Daniel Vetter wrote: > > Hi all, > > You might have read the short take in the X.org board meeting minutes > already, here's the long version. > > The good news: gitlab.fd.o has become very popular with our > communities, and is used extensively. This especially includes all the > CI integration. Modern development process and tooling, yay! > > The bad news: The cost in growth has also been tremendous, and it's > breaking our bank account. With reasonable estimates for continued > growth we're expecting hosting expenses totalling 75k USD this year, > and 90k USD next year. With the current sponsors we've set up we can't > sustain that. We estimate that hosting expenses for gitlab.fd.o > without any of the CI features enabled would total 30k USD, which is > within X.org's ability to support through various sponsorships, mostly > through XDC. > > Note that X.org does no longer sponsor any CI runners themselves, > we've stopped that. The huge additional expenses are all just in > storing and serving build artifacts and images to outside CI runners > sponsored by various companies. A related topic is that with the > growth in fd.o it's becoming infeasible to maintain it all on > volunteer admin time. X.org is therefore also looking for admin > sponsorship, at least medium term. > > Assuming that we want cash flow reserves for one year of gitlab.fd.o > (without CI support) and a trimmed XDC and assuming no sponsor payment > meanwhile, we'd have to cut CI services somewhere between May and June > this year. The board is of course working on acquiring sponsors, but > filling a shortfall of this magnitude is neither easy nor quick work, > and we therefore decided to give an early warning as soon as possible. > Any help in finding sponsors for fd.o is very much appreciated. Some clarification I got from Daniel in a private conversation, since I was confused about what the money was paying for exactly: We're paying 75K USD for the bandwidth to transfer data from the GitLab cloud instance. i.e., for viewing the https site, for cloning/updating git repos, and for downloading CI artifacts/images to the testing machines (AFAIU). I was not aware that we were being charged for anything wrt GitLab hosting yet (and neither was anyone on my team at Intel that I've asked). This... kind of needs to be communicated. A consistent concern put forth when we were discussing switching to GitLab and building CI was... how do we pay for it. It felt like that concern was always handwaved away. I heard many times that if we needed more runners that we could just ask Google to spin up a few more. If we needed testing machines they'd be donated. No one mentioned that all the while we were paying for bandwidth... Perhaps people building the CI would make different decisions about its structure if they knew it was going to wipe out the bank account. What percentage of the bandwidth is consumed by transferring CI images, etc? Wouldn't 75K USD would be enough to buy all the testing machines we need and host them within Google or wherever so we don't need to pay for huge amounts of bandwidth? I understand that self-hosting was attractive so that we didn't find ourselves on the SourceForge-equivalent hosting platform of 2022, but is that risk real enough to justify spending 75K+ per year? If we were hosted on gitlab.com or github.com, we wouldn't be paying for transferring CI images to CI test machines, etc, would we? So what do we do now? Have we painted ourselves into a corner? ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] gitlab.fd.o financial situation and impact on services
Hi all, You might have read the short take in the X.org board meeting minutes already, here's the long version. The good news: gitlab.fd.o has become very popular with our communities, and is used extensively. This especially includes all the CI integration. Modern development process and tooling, yay! The bad news: The cost in growth has also been tremendous, and it's breaking our bank account. With reasonable estimates for continued growth we're expecting hosting expenses totalling 75k USD this year, and 90k USD next year. With the current sponsors we've set up we can't sustain that. We estimate that hosting expenses for gitlab.fd.o without any of the CI features enabled would total 30k USD, which is within X.org's ability to support through various sponsorships, mostly through XDC. Note that X.org does no longer sponsor any CI runners themselves, we've stopped that. The huge additional expenses are all just in storing and serving build artifacts and images to outside CI runners sponsored by various companies. A related topic is that with the growth in fd.o it's becoming infeasible to maintain it all on volunteer admin time. X.org is therefore also looking for admin sponsorship, at least medium term. Assuming that we want cash flow reserves for one year of gitlab.fd.o (without CI support) and a trimmed XDC and assuming no sponsor payment meanwhile, we'd have to cut CI services somewhere between May and June this year. The board is of course working on acquiring sponsors, but filling a shortfall of this magnitude is neither easy nor quick work, and we therefore decided to give an early warning as soon as possible. Any help in finding sponsors for fd.o is very much appreciated. Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/mesa-dev
