Hi, -0 for this idea. I don't strongly against this idea but I don't like this idea.
The current raw GitHub Actions configurations + raw Docker tools (+ raw shell scripts) are simple and easy to understand for many developers because there are no magics. If we create our tools, they will introduce some magics. It may block new developers. (We want to increase Apache Arrow developers as much as possible, right?) If we use raw GitHub Actions configuration, we can introduce knowledge from other projects. For example, I'll migrate MinGW builds on AppVeyor to GitHub Actions. I know how to setup MinGW environment on GitHub Actions: https://github.com/groonga/groonga/blob/master/.github/workflows/windows-mingw.yml If I can use raw GitHub Actions configuration, I'll re-use the similar approach in Apache Arrow easily. If we have our tools, I need to translate my GitHub Actions configuration knowledge to codes/configurations for our tools. I agree with duplication. But it may be resolved by improving GitHub Actions configuration syntax. For example, GitLab CI configuration has "include" feature: https://docs.gitlab.com/ee/ci/yaml/#include If GitHub Actions configuration implements similar feature, duplication will be able to reduced. GitHub accepts feature requests in community forum: https://github.community/t5/GitHub-Actions/bd-p/actions Feature request example: https://github.community/t5/GitHub-Actions/Feature-Request-context-property-similar-to-github-repository/m-p/38964/highlight/true#M3608 No "include" like feature in GitHub Actions: https://github.community/t5/GitHub-Actions/Is-it-possible-to-reuse-workflow-yaml-to-setup-similar-workflows/m-p/40634# Thanks, -- kou In <CAHM19a4JFB1G_bOKREk-wxCc=dh34qt3kq53s1emwqpfwwc...@mail.gmail.com> "Re: [DISCUSS] Improving GitHub Actions configurations + tooling" on Tue, 4 Feb 2020 22:54:14 +0100, Krisztián Szűcs <[email protected]> wrote: > On Tue, Feb 4, 2020 at 6:31 PM Wes McKinney <[email protected]> wrote: >> >> I'm personally not too concerned with the details as long as people >> generally agree that the solution is maintainable (Kou's and Antoine's >> feedback here would be helpful) and there is not an abundant odor of >> code duplication > I'm not entirely satisfied with the current solution, but so far so good. > > Before improving the current GHA setup I'd be indeed nice to see others' > opinions and preferences. >> >> On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson >> <[email protected]> wrote: >> > >> > What if we wrote our own action(s) to wrap up some of the boilerplate? It >> > doesn't seem that there are any off-the-shelf actions we could use to drive >> > docker-compose: >> > https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose >> > but I don't think it would be that difficult to wrap `docker-compose pull >> > $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in >> > an action. >> > >> > Neal >> > >> > On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <[email protected]> >> > wrote: >> > >> > > Hi, >> > > >> > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <[email protected]> wrote: >> > > > >> > > > hi folks, >> > > > >> > > > I have noticed that many of our GitHub Actions configurations are very >> > > > similar to each other >> > > They are indeed. >> > > > >> > > > https://www.diffchecker.com/eF4tHdzo >> > > > >> > > > Aside from the "copy-paste" issue, some work would have to be done to >> > > > generate a Crossbow configuration using GHA. >> > > Do you mean having GHA as a backend for crossbow? @Kou has just >> > > added support for that in [1] >> > > > >> > > > It seems like a solution to these issues is to create a program to >> > > > generate the GHA configurations (using some templates or other tools). >> > > > So what is in .github/workflows would not be edited by human hands in >> > > > general but rather generated by this program. >> > > That would be quite similar to what I've implemented in ursabot [2], just >> > > generating GHA flavored ymls instead of buildbot objects, so it seems >> > > doable. Of course we'll need commit hooks to force the regeneration of >> > > these configuration files. >> > > > >> > > > This program could also assist with local automation for >> > > > reproducibility purposes (for example, locally executing a cascade of >> > > > dependent docker-compose steps). >> > > Another independent improvement could be to ditch docker-compose >> > > completely. >> > > I'd say that 70% of the docker-compose.yml [3] and the relating >> > > dockerfiles are >> > > filled with duplications necessary because of the limited parametrization >> > > and >> > > reusability of docker and docker-compose. It also makes harder to use new >> > > docker >> > > features like https://docs.docker.com/buildx/working-with-buildx/ >> > > >> > > Again I'm referring ursabot where I've already implemented the ideas, the >> > > docker files [4] and the image hierarchy from the compose file [3] could >> > > be >> > > replaced by something similar like the ursabot docker utility [6]. >> > > The builder definitions [7] which are the counterparts for the scripts >> > > [8] compose >> > > command [9] and compose parameters [10]. >> > > I'm not saying that we should use the exact build definition syntax >> > > from ursabot, >> > > we can have arbitrary ymls etc. but the configuration I've referenced >> > > from >> > > ursabot approximates what we can achieve by increasing the abstraction >> > > level. >> > > >> > > Note, that the ursabot sources I've referenced are not GPL contaminated. >> > > > >> > > > Thoughts? >> > > On the other hand I see one big advantage of the current setup. By using >> > > plain >> > > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small >> > > bash >> > > scripts, our CI configuration is more idiomatic. It sits closer to the >> > > developers' >> > > expectations. >> > > It's hard for me to judge the developer friendliness of either of "CI >> > > tools" >> > > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator >> > > of those, but I suppose that the new GHA setup is the easiest to work >> > > with. >> > > > >> > > Thanks, Krisztian >> > > >> > > [1] https://github.com/apache/arrow/pull/6286 >> > > [2] >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256 >> > > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml >> > > [4] https://github.com/apache/arrow/tree/master/ci/docker >> > > [6] >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py >> > > [7] >> > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332 >> > > [8] https://github.com/apache/arrow/tree/master/ci/scripts >> > > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306 >> > > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264 >> > > > - Wes >> > >
