It sounds like this would be a good reason to use BuildKite, which AFAIU can automatically provision and operate cloud resources for us?
Le 04/03/2020 à 16:21, Wes McKinney a écrit : > hi folks, > > The tornado the night before last in Nashville, Tennessee temporarily > disabled the physical hardware that I have been running there where > we've been running "Ursabot" builds and where we've been experimenting > with other self-hosted CI solutions like GitHub Actions Self-Hosted > Runners and Buildkite. > > While dedicated physical hardware can be useful to reduce cloud > computing costs, I think this natural disaster should help inform our > approach to this problem: > > * In the event that physical hosted infrastructure becomes > unavailable, we eventually should have the capability to spin up > machines in the cloud with the desired properties (GCE provides both > Linux and Windows VMs, for example) > * Adding new machines to our CI process ideally should not require a > human in-the-loop (GHA presently requires a human -- in particular > someone from ASF Infra -- in the loop to add workers, so this IMHO > should be taken into consideration) > > Any other thoughts about this topic would be welcome. > > Thanks > Wes > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs > <szucs.kriszt...@gmail.com> wrote: >> >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <wesmck...@gmail.com> wrote: >>> >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs >>> <szucs.kriszt...@gmail.com> wrote: >>>> >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <wesmck...@gmail.com> wrote: >>>>> >>>>> hi Ganesh, >>>>> >>>>> Thanks for writing. >>>>> >>>>> I've been working on setting up Buildkite (BK) as a way for third >>>>> parties for attach machines to run builds on, with a free organization >>>>> at >>>>> >>>>> https://buildkite.com/apache-arrow >>>>> >>>>> Configuring a new machine to accept builds is very easy [1] and takes >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work >>>>> on Windows). Currently I've attached 6 machines: >>>>> >>>>> * 2 CUDA-capable Linux x86 >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable >>>>> * 1 macOS >>>>> >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds >>>>> triggered in BK can report commit statuses on GitHub [2] >>>>> >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this >>>>> also but the workflow for new machines to be contributed needs to be >>>>> proven out. >>>> I've already tried it out, and setting up self-hosted github runners is >>>> just as >>>> easy as with buildkite, drawbacks: >>> >>> I don't mean to be argumentative, but I don't see how this can be true >>> if we don't have access to the "Settings" tab on GitHub >> On a fork where I have access for that tab. >>> >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners >>> >>>> - I'm unsure how would the tagging selection work in practice [1] >>>> - We won't have access to the runners dashboard in lack of admin rights >>>> for the apache/arrow repository - so we need to test out the workflow. >>>> >>>> I've created an INFRA ticket to get some information and to track it: >>>> https://issues.apache.org/jira/browse/INFRA-19875 >>>> >>>> [1] >>>> https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner >>>>> >>>>> Thanks, >>>>> Wes >>>>> >>>>> [1]: >>>>> https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217 >>>>> >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ganesh.r...@linaro.org> >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> I am following up on the discussion from here >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have >>>>>> dedicated arm hardware for CI setup. We can surely help with that if we >>>>>> get >>>>>> a go-ahead from the project. >>>>>> >>>>>> Thanks, >>>>>> Ganesh >>>>>> >>>>>> -- >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
