On Fri, Jun 14, 2019 at 9:04 PM Wes McKinney <wesmck...@gmail.com> wrote:
> hi Krisz, > > Thanks for working on this! It already helped me fix a Python 2.7-only > bug yesterday https://github.com/apache/arrow/pull/4553 > > I have a bunch of questions: > > * What is the license of the ursabot codebase? Seems like it could be > GPL if Buildbot itself is [2] and you have reused any Buildbot code. > This is not mentioned in the README so I think you need to call this > out and advise people to be careful that any code contributed to this > codebase is stuck in GPL-land. Is it possible to separate the work > you've done from any GPL-ness? I don't think you can be too paranoid > about this kind of thing, and the longer you wait to draw a clear line > around any contaminated code the harder it will be to disentangle. > Created an issue for it https://github.com/ursa-labs/ursabot/issues/105 > > * How brittle is the Buildbot master? It currently resides in my home, > but what if a natural disaster (like [1] from 2010) occurs in > Nashville causing an extended power outage (or a temporary outage > requiring human intervention while I'm out of town)? Is the Buildbot > master state backed up? Can it be easily migrated to a new host > (Dockerized, even)? Either way we need a contingency plan. > It doesn't have a backup yet, although it only matters for historical reasons. If I prune its database and restart the buildmaster nothing changes except the losing the previous builds' result - so everything will work the same way as before erasing the database. It can be pretty easily migrated to another host, besides setting up the networking with the workers and a database server (or sqlite) nothing more esoteric is required. > > * The availability of Buildbot suggests we should try to decouple our > CI procedures from anything Travis CI specific and use Docker instead, > at least for the Linux builds. This has the side benefit of enabling > contributors to reproduce CI build locally. Can you create some JIRAs > about this? > Yes, IMO docker is preferable, even for Windows containers [1]. Will do, but I'm curious for other's opinion too. @Uwe? > > * Currently there are only Linux-based x86 and Aarch64 builders. Given > the periodic bandwidth issues with Windows on Appveyor, getting more > builds off of Appveyor would help us all. If I wanted to set up a new > machine (including a Windows-based one), how do I do that? > We don't have windows builders yet, so it'll require some (not much) development. Theoretically we can either use windows workers running within windows docker containers or directly on the windows host. After having a windows worker connected to the buildmaster We need to setup builders which are able to run on windows (this is mostly about converting the build steps to windows specific ones [2], although it might work out of the box - haven't tested it yet so cannot say much more about it). > > * Do machines have to be co-located on the same physical network as > the master, or can they reside in other locations? > It is preferable to have a master in the same network where the workers are, because the build steps are rpc calls made by the master. In order to scale our cluster we can have a multi-master setup [3], so spining up workers at different locations is possible. For example we could use machines from public cloud providers to offload the current travis/appveyor builds (with better availability then your/Wes' network). [1]: https://www.docker.com/products/windows-containers [2]: https://github.com/ursa-labs/ursabot/blob/master/ursabot/builders.py#L420 [3]: http://docs.buildbot.net/2.3.1/full.html#multi-master-mode > Thanks, > Wes > > [1]: https://en.wikipedia.org/wiki/2010_Tennessee_floods > [2]: https://github.com/buildbot/buildbot > > On Fri, Jun 14, 2019 at 9:48 AM Krisztián Szűcs > <szucs.kriszt...@gmail.com> wrote: > > > > Hello All, > > > > We're developing a buildbot application to utilize Ursa Labs’ > > physical machines called Ursabot. Buildbot [1] is used by > > major open source projects, like CPython and WebKit [2]. > > > > The source code is hosted at [3], the web interface is > > accessible at [4]. The repository contains a short guide > > about the goals, implementation and the interfaces we can > > drive ursabot. The most notable way to trigger ursabot builds > > is via sending github comments mentioning @ursabot machine > > account, for more see [5]. > > > > Currently we have builders for the C++ implementation and > > the Python bindings on AMD64 and ARM64 architectures. > > It is quite easy to attach workers to the buildmaster [7], so > > We can scale our build cluster to test and run on-demand builds > > (like benchmarks, packaging tasks) on more platforms. > > > > Yesterday we've enabled the github status push reporter > > to improve the visibility of ursabot, although we were testing > > the builders in the last couple of weeks. I hope no one has > > a hard objection against this new CI. Arrow has already started > > to outgrow Travis-CI and Appveyor's capacity and we're trying > > to make the build system quicker and more robust. > > > > Please don't hesitate to ask any questions! > > > > Thanks, Krisztian > > > > [1]: http://buildbot.net/ > > [2]: https://github.com/buildbot/buildbot/wiki/SuccessStories > > [3]: https://github.com/ursa-labs/ursabot > > [4]: https://ci.ursalabs.org > > [5]: https://github.com/ursa-labs/ursabot#driving-ursabot > > [7]: https://github.com/ursa-labs/ursabot/blob/master/default.yaml#L115 >
