Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Gavin McDonald
Hi , Just to note I have emailed DockerHub, asking for clarification on our account and what our benefits are. On Thu, Oct 29, 2020 at 6:34 PM Allen Wittenauer wrote: > > > On Oct 29, 2020, at 9:21 AM, Joan Touzet wrote: > > > > (Sidebar about the script's details) > > Sure. > > > I t

Builds related Meeting

2020-10-29 Thread Gavin McDonald
Hi All, I had a thought that it might be an interesting idea to have a meeting space where folks from this list can chat with each other and Infra about all things ASF CI. It is also a chance to informally meet some members of the Infra Team. The meeting agenda is open and everyone is welcome to

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Allen Wittenauer
> On Oct 29, 2020, at 9:21 AM, Joan Touzet wrote: > > (Sidebar about the script's details) Sure. > I tried to read the shell script, but I'm not in the headspace to fully parse > it at the moment. If I'm understanding correctly, this will still catch > CouchDB's CI docker images if

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Joan Touzet
On 2020-10-29 11:37 a.m., Allen Wittenauer wrote: On Oct 28, 2020, at 11:57 PM, Chris Lambertus wrote: Infra would LOVE a smarter way to clean the cache. We have to use a heavy hammer because there are 300+ projects that want a piece of it, and who don’t clean up.. We are not build enginee

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Allen Wittenauer
> On Oct 28, 2020, at 11:57 PM, Chris Lambertus wrote: > > Infra would LOVE a smarter way to clean the cache. We have to use a heavy > hammer because there are 300+ projects that want a piece of it, and who don’t > clean up.. We are not build engineers, so we rely on the community to advise

Re: Jenkins Credential that can comment on GitHUB PRs

2020-10-29 Thread Matt Sicker
I can almost guarantee that credential you found is someone’s personal access token anyways. I don’t think anything has been set up org wide besides the GitHub app for webhooks and checkouts. On Thu, Oct 29, 2020 at 02:10 Istvan Toth wrote: > There was quite a bit of traffic on this list about h

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Vladimir Sitnikov
Just my 2c, GitHub team is working with DockerHub, and GitHub-based runners won't be affected by DockerHub limits. Of course, Travis-based CI jobs and ASF-based CI jobs might still be affected. See https://github.com/actions/virtual-environments/issues/1445#issuecomment-713861495 Vladimir

Re: GitHub Actions Concurrency Limits for Apache projects

2020-10-29 Thread Jarek Potiuk
Hello, Together with Tobiasz we've just merged and started to test extensively the optimization in Airflow. It works slightly differently than initially planned - we came up with a few more optimizations. It turned out that in most cases we actually do not need to run full matrix builds and we can

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Jarek Potiuk
And also another comment, If this is a concern only for DockerHub as a "CI" workflow, for now, I can actually recommend the Github Docker Registry (even if it is technically deprecated). It does a great job for us at Airflow. We build our images during CI only once and push them to the Github Reg

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Jarek Potiuk
> > > As I mentioned in the referenced ticket, the Github Container Registry > (“GCR”) is not something Infra can immediately support due to Github’s > limitations. The Board collectively with Infra needs to determine how or if > the GCR fits our needs from both operational and fiscal perspectives.

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Chris Lambertus
> On Oct 29, 2020, at 12:19 AM, Jarek Potiuk wrote: > > Got the message too and also got very concerned. > > There is an easy (I think) solution though. The new GitHub Container > Registry is a good replacement, providing that it is enabled by INFRA. I've > opened the ticket few weeks ago as

Re: Docker rate limits likely spell DOOM for any Apache project CI workflow relying on Docker Hub

2020-10-29 Thread Jarek Potiuk
Got the message too and also got very concerned. There is an easy (I think) solution though. The new GitHub Container Registry is a good replacement, providing that it is enabled by INFRA. I've opened the ticket few weeks ago as we would love to switch to it from the current (deprecated) GitHub Do

Jenkins Credential that can comment on GitHUB PRs

2020-10-29 Thread Istvan Toth
There was quite a bit of traffic on this list about having a credential that is able to comment on GitHub PRs, but we (Phoenix project) still have this problem. What I am trying to do is posting a comment on the PR like this: https://github.com/apache/phoenix/pull/940#issuecomment-71308 (Deve