I've trimmed down the number of triggered builds on pull requests by converting them to run on master only or cron builds. Alsoe added the action filters including the changed path patterns. I've also collected ~30 follow up JIRAs aggregating the problems I came across during the refactor and possible further improvements and optimizations like caching.
The PR is ready to be merged [1]. I expect multiple issues after merging the PR, despite that I was trying to port everything thoroughly. The changeset is simply too big, but we need to start somewhere, and sooner is better. [1]: https://github.com/apache/arrow/pull/5589 On Thu, Oct 31, 2019 at 10:21 PM Wes McKinney <wesmck...@gmail.com> wrote: > > hi Krisz -- I just left comments on the PR. This definitely looks like > a good step forward. My main comment is that I think there are too > many C++/Python tasks to run on an each-commit basis. Ideally many of > these would be run nightly. There is also a certain amount of > redundancy in rebuilding the C++ library multiple times before running > each dependent set of tests, whereas in Travis we build the C++ > library once then test both C++ and Python. If there is sufficient > number of builders then perhaps it doesn't matter so much > > It seems there are a few things, like action filtering (similar to > "detect-changes.py") based on what was changed that would need to get > done before this can be merged. > > - Wes > > On Fri, Oct 25, 2019 at 7:25 PM Krisztián Szűcs > <szucs.kriszt...@gmail.com> wrote: > > > > Hi, > > > > During the release of 0.15.1-RC0 I literally had to wait days > > to ensure that the Travis, Appveyor and Crossbow builds > > were all passing for the release branch. Additionally each > > newly added patch was delaying the process by 8 hrs or so > > (actually felt like 16). > > > > Recently I've been working on to incorporate the advantages > > of the Buildbot setup into our current docker-compose > > configuration, including support for multiple architectures > > and platforms, reusing docker images and caching dependency > > installation steps. It tries to follow the semantics of ursabot, > > but using only docker-compose and tiny shell scripts. > > > > This refactoring also includes GitHub Actions workflows for > > Windows and macOS as well, reusing the same (bash) builds > > scripts. The docker configuration and the scripts are CI agnostic. > > Last but not least, I've managed to clean up a lot of things > > including every travis builds, and three Appveyor builds. > > As an example the ci [3] and dev [4] folders got much cleaner. > > > > The majority of the builds are passing [2], but due to the size > > of the pull request [1] reviews for relevant workflows like the > > JavaScript, C#, Rust, JNI, etc. would be much appreciated. > > I'll be on vacation until Wednesday, but will try to respond on > > both GH and the ML. > > > > Thanks, Krisztian > > > > [1]: https://github.com/apache/arrow/pull/5589 > > [2]: https://github.com/apache/arrow/runs/275685241 > > [3]: > > https://github.com/apache/arrow/tree/9c7e7289b9c9486c13a02e7cb5682a0f9f274ec6/ci > > [4]: > > https://github.com/apache/arrow/tree/9c7e7289b9c9486c13a02e7cb5682a0f9f274ec6/dev
