Hi folks, First off, I'd like to say that I wasn't such warm replies for help regarding our CI migration, so thank you Chris and Jarek for your enthusiastic comments.
As such, I hadn't really thought about how to collaborate on getting the new config together :) Now that I've started it, I won't mind using https://github.com/cs-shadow/buildstream as a staging ground for prepping our changes before we officially move under github/apache. I'm also up for any alternatives in case someone has a better idea. If you could send me your github usernames, I can add you as collaborators on my fork. PRs are also welcome. This also applies to anyone else who is also interested. Later today or tomorrow, I plan to finish pre-merge checks and create some issues for what's left so that people can see what's left to do. In the meantime, windows and overnight tests are wide open to migrate in case anyone is interested. Some comments inline. On Tue, Aug 18, 2020 at 2:31 PM Jarek Potiuk <[email protected]> wrote: > > Hey Chris, > > As mentioned before - i'd love to help bringing my Apache Airflow > experiences with GitHub. As you commented - I would also love to have a PR > out of the code (even draft)? I have some comments that might be best if > they are referring to the actual code and happy to add more comments :). > > Thanks for the notes Chandan, it makes it really easy to follow. There are > > 3 things that I can have a prod at: > > > > - "GitHub Actions do not support YAML variables and/or YAML anchors, which > > leads to some repetition." Yeh I think that this is a problem. So the > > way buildbarn approach this is to use jsonnet to generate the github > > actions, which I think is an effective route > > https://github.com/buildbarn/bb-storage/commit/41a7e29dfebb625729a0c4f0e8285a254781e698#diff-02d9c370a663741451423342d5869b21 > > I'm happy to experiement with such an approach and report back. > > > > As of last week GitHub actions support Composite Run steps: > https://github.blog/changelog/2020-08-07-github-actions-composite-run-steps/ > that > might do what you want to do regarding code reuse. > > Comment: I also missed anchors initially, but over time I think the > Composite Run steps are a much better solution if you have quite a number > of jobs to write. The problem with anchors and Yaml variables is that they > are designed to make the yaml more DRY. But I think it is much more > important that the CI code is easy to read and that you know what's going > on when you look at the step in isolation rather than having to refer to > something defined in another part of Yaml (readability trumps > orthogonality). The way how anchors are merged and their syntax is rather > weird IMHO and I think CI yamls should be (similarly as tests) much more > DAMP than DRY ( > https://stackoverflow.com/questions/6453235/what-does-damp-not-dry-mean-when-talking-about-unit-tests > ) I can see where you're coming from. I started missing anchors even before I started writing the yaml file when I read that they aren't allowed. This was mostly because I was trying to map Actions concepts 1:1 to GitLab CI concepts. But, after I started writing it, I felt like they could be avoided in most cases. In our case, the matrix job seems good enough for the most part. > > - "persistent artifact cache" So I know that there have been efforts in > > this area via > > https://gitlab.com/BuildStream/buildstream/-/merge_requests/1997 (now > > merged) and more recently with the asset cache at > > https://gitlab.com/BuildStream/infrastructure/infrastructure/-/merge_requests/2. > > It's probably worth coordinating with the authors of both MRs to smooth the > > transition. > > Not sure what is the "cache" here. But GA has excellent support for caching > artifacts including API to query and retrieve them. We are using it > extensively in Airflow - we publish generated packages, logs, > documentation, and much more this way: > https://github.com/apache/airflow/actions/runs/213556233 I can briefly elaborate what we mean by a cache in this context. For our integration tests, we need to fetch some sources from various locations. These operations can take a bit of time so we cache them in our CI setup to save time. We want this to persist across runs. I suspect https://github.com/actions/cache is what we want. > > On a broader point, is there anywhere we can track and review each other's > > work? I'm happy to work off https://github.com/cs-shadow/buildstream but > > other locations work for me. As mentioned above, let's use that unless anyone has a better idea. > I also looked up some earlier discussions in the thread and here are my > comments: > > Since we will have a public repository, we qualify for "free" and > > "unlimited" actions. I am sure both those terms come with associated > > fineprint. So far, it seems like Actions could cope up with ~4-5 > > pipelines in parallel (each with 7-8 jobs). I am not sure whether the > > free tier will be able to keep up with our load once we start getting > > real patches on GitHub. If not, we may need to add some custom runners > > (similar to what we have on GitLab). > > > Apache has Enterprise level of support. Once I saw a message about it, but > I cannot find the place any more. But when you raise a ticket and choose > "Apache" as organisation you can see it is "Enterprise level". BTW. They > are usually very helpful and respond in under 2 hours usually. Happy to hear that. > Out of the trenches: > > We run a lot of huge builds in Apache Airflow, I think when we run on > Travis we were the 3rd or 4th highest-use Apache project in Travis CI - > right after Apache Kafka I believe and it only grew 2-3 times since then. > We have ~10000 builds now and counting over the last ~4 months. And at > least 1/3 of those builds are of this kind ( > https://github.com/apache/airflow/actions/runs/213556233) - 30 jobs, each > using 10-30 minutes of machine time. Which at the very least is > 10x30x10000/3/4/60 ~ 5000 hours of build hours per month. That's a lot. We > are still optimising it because we want to be good citizens though and I am > finishing a change to decrease it by 20/30%. > > And indeed - you can set up private workers on Azure/Amazon/GCP if needed. > It's super easy and you can even setup auto-scaling self-hosted runners if > you need AWS example here: > https://040code.github.io/2020/05/25/scaling-selfhosted-action-runners and > different options to deploy on GCP : > https://github.blog/2020-08-04-github-actions-self-hosted-runners-on-google-cloud/ > (auto-scaling > with AppEngine, Kubernetes, persistent, even multi-cloud with Anthos if you > want). I'm hoping that we won't actually have to manage runners ourselves, and that the GitHub runners will be enough. Let's see! Cheers, Chandan
