On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <anto...@python.org> wrote: > > > It sounds like this would be a good reason to use BuildKite, which AFAIU > can automatically provision and operate cloud resources for us? That's true for buildbot as well [1] including openstack and mesos support which would be nice for physical machines.
With the ursa machines down we ended up with two missing services: - nightly trigger and report: I've already ported the cron jobs triggering the builds and reporting to github actions [2] - the comment bot: it is implemented in ursabot including the github event listener, so buildkite wouldn't provide a solution here because we need to maintain a web service for it. The solution for the comment bot could be to factor out the implementation from ursabot and run it from a github actions build triggered using the issue_comment event [3] (buildkite doesn't seem to support it currently [4]). We can also host the buildbot buildmaster in the cloud, at least until we decomission all of its services. [1] https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers [2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows [3] https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment [4] https://github.com/buildkite/feedback/issues/288 > > > Le 04/03/2020 à 16:21, Wes McKinney a écrit : > > hi folks, > > > > The tornado the night before last in Nashville, Tennessee temporarily > > disabled the physical hardware that I have been running there where > > we've been running "Ursabot" builds and where we've been experimenting > > with other self-hosted CI solutions like GitHub Actions Self-Hosted > > Runners and Buildkite. > > > > While dedicated physical hardware can be useful to reduce cloud > > computing costs, I think this natural disaster should help inform our > > approach to this problem: > > > > * In the event that physical hosted infrastructure becomes > > unavailable, we eventually should have the capability to spin up > > machines in the cloud with the desired properties (GCE provides both > > Linux and Windows VMs, for example) > > * Adding new machines to our CI process ideally should not require a > > human in-the-loop (GHA presently requires a human -- in particular > > someone from ASF Infra -- in the loop to add workers, so this IMHO > > should be taken into consideration) > > > > Any other thoughts about this topic would be welcome. > > > > Thanks > > Wes > > > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs > > <szucs.kriszt...@gmail.com> wrote: > >> > >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <wesmck...@gmail.com> wrote: > >>> > >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs > >>> <szucs.kriszt...@gmail.com> wrote: > >>>> > >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <wesmck...@gmail.com> > >>>> wrote: > >>>>> > >>>>> hi Ganesh, > >>>>> > >>>>> Thanks for writing. > >>>>> > >>>>> I've been working on setting up Buildkite (BK) as a way for third > >>>>> parties for attach machines to run builds on, with a free organization > >>>>> at > >>>>> > >>>>> https://buildkite.com/apache-arrow > >>>>> > >>>>> Configuring a new machine to accept builds is very easy [1] and takes > >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work > >>>>> on Windows). Currently I've attached 6 machines: > >>>>> > >>>>> * 2 CUDA-capable Linux x86 > >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable > >>>>> * 1 macOS > >>>>> > >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds > >>>>> triggered in BK can report commit statuses on GitHub [2] > >>>>> > >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this > >>>>> also but the workflow for new machines to be contributed needs to be > >>>>> proven out. > >>>> I've already tried it out, and setting up self-hosted github runners is > >>>> just as > >>>> easy as with buildkite, drawbacks: > >>> > >>> I don't mean to be argumentative, but I don't see how this can be true > >>> if we don't have access to the "Settings" tab on GitHub > >> On a fork where I have access for that tab. > >>> > >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners > >>> > >>>> - I'm unsure how would the tagging selection work in practice [1] > >>>> - We won't have access to the runners dashboard in lack of admin rights > >>>> for the apache/arrow repository - so we need to test out the workflow. > >>>> > >>>> I've created an INFRA ticket to get some information and to track it: > >>>> https://issues.apache.org/jira/browse/INFRA-19875 > >>>> > >>>> [1] > >>>> https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner > >>>>> > >>>>> Thanks, > >>>>> Wes > >>>>> > >>>>> [1]: > >>>>> https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh > >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217 > >>>>> > >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ganesh.r...@linaro.org> > >>>>> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> I am following up on the discussion from here > >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have > >>>>>> dedicated arm hardware for CI setup. We can surely help with that if > >>>>>> we get > >>>>>> a go-ahead from the project. > >>>>>> > >>>>>> Thanks, > >>>>>> Ganesh > >>>>>> > >>>>>> -- > >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
