On Tue, Feb 4, 2020 at 6:31 PM Wes McKinney <[email protected]> wrote: > > I'm personally not too concerned with the details as long as people > generally agree that the solution is maintainable (Kou's and Antoine's > feedback here would be helpful) and there is not an abundant odor of > code duplication I'm not entirely satisfied with the current solution, but so far so good.
Before improving the current GHA setup I'd be indeed nice to see others' opinions and preferences. > > On Tue, Feb 4, 2020 at 9:48 AM Neal Richardson > <[email protected]> wrote: > > > > What if we wrote our own action(s) to wrap up some of the boilerplate? It > > doesn't seem that there are any off-the-shelf actions we could use to drive > > docker-compose: > > https://github.com/marketplace?utf8=%E2%9C%93&type=actions&query=docker-compose > > but I don't think it would be that difficult to wrap `docker-compose pull > > $JOB && docker-compose build $JOB && docker-compose run $JOB` or similar in > > an action. > > > > Neal > > > > On Tue, Feb 4, 2020 at 6:57 AM Krisztián Szűcs <[email protected]> > > wrote: > > > > > Hi, > > > > > > On Mon, Feb 3, 2020 at 5:25 PM Wes McKinney <[email protected]> wrote: > > > > > > > > hi folks, > > > > > > > > I have noticed that many of our GitHub Actions configurations are very > > > > similar to each other > > > They are indeed. > > > > > > > > https://www.diffchecker.com/eF4tHdzo > > > > > > > > Aside from the "copy-paste" issue, some work would have to be done to > > > > generate a Crossbow configuration using GHA. > > > Do you mean having GHA as a backend for crossbow? @Kou has just > > > added support for that in [1] > > > > > > > > It seems like a solution to these issues is to create a program to > > > > generate the GHA configurations (using some templates or other tools). > > > > So what is in .github/workflows would not be edited by human hands in > > > > general but rather generated by this program. > > > That would be quite similar to what I've implemented in ursabot [2], just > > > generating GHA flavored ymls instead of buildbot objects, so it seems > > > doable. Of course we'll need commit hooks to force the regeneration of > > > these configuration files. > > > > > > > > This program could also assist with local automation for > > > > reproducibility purposes (for example, locally executing a cascade of > > > > dependent docker-compose steps). > > > Another independent improvement could be to ditch docker-compose > > > completely. > > > I'd say that 70% of the docker-compose.yml [3] and the relating > > > dockerfiles are > > > filled with duplications necessary because of the limited parametrization > > > and > > > reusability of docker and docker-compose. It also makes harder to use new > > > docker > > > features like https://docs.docker.com/buildx/working-with-buildx/ > > > > > > Again I'm referring ursabot where I've already implemented the ideas, the > > > docker files [4] and the image hierarchy from the compose file [3] could > > > be > > > replaced by something similar like the ursabot docker utility [6]. > > > The builder definitions [7] which are the counterparts for the scripts > > > [8] compose > > > command [9] and compose parameters [10]. > > > I'm not saying that we should use the exact build definition syntax > > > from ursabot, > > > we can have arbitrary ymls etc. but the configuration I've referenced from > > > ursabot approximates what we can achieve by increasing the abstraction > > > level. > > > > > > Note, that the ursabot sources I've referenced are not GPL contaminated. > > > > > > > > Thoughts? > > > On the other hand I see one big advantage of the current setup. By using > > > plain > > > GHA ymls, a single docker-compose.yml, Dockerfiles and a set of small bash > > > scripts, our CI configuration is more idiomatic. It sits closer to the > > > developers' > > > expectations. > > > It's hard for me to judge the developer friendliness of either of "CI > > > tools" > > > (GHA+docker setup, crossbow, ursabot) because I was the main perpetrator > > > of those, but I suppose that the new GHA setup is the easiest to work > > > with. > > > > > > > Thanks, Krisztian > > > > > > [1] https://github.com/apache/arrow/pull/6286 > > > [2] > > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/master.cfg#L67-L256 > > > [3] https://github.com/apache/arrow/blob/master/docker-compose.yml > > > [4] https://github.com/apache/arrow/tree/master/ci/docker > > > [6] > > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/docker.py > > > [7] > > > https://github.com/ursa-labs/ursabot/blob/master/projects/arrow/arrow/builders.py#L332 > > > [8] https://github.com/apache/arrow/tree/master/ci/scripts > > > [9] https://github.com/apache/arrow/blob/master/docker-compose.yml#L306 > > > [10] https://github.com/apache/arrow/blob/master/docker-compose.yml#L264 > > > > - Wes > > >
