Yeah, there're unfortunately good reasons not to have any accounts with
write permission in the GitHub repo. It can cause all sorts of problems if
anything were actually pushed. It also allows lots of other things like
editing other people's comments. GitHub should really separate that out for
the purpose of bots with minimum required access anyway. But yeah, without
write, it's a no-go. Comments are a very reasonable alternative, though.
It's definitely worth a few minutes to get things set up on the ASF Jenkins
if those ubuntu slaves have the requirements. As it stands, the only
requirements for a build are:
- git
- bash
- docker
- docker-compose
- which requires python
I believe the first three are satisfied by the build hosts. I don't know
that docker-compose is available. It's 100% worth finding out, though.
If it's not, we can do one of:
- Run docker-compose inside a docker instance with the docker port
forwarded into the container to allow it to manage sibling containers.
- Re-create the subset of docker-compose behaviour that we actually use
in a build script.
- Give up.
I mention the first option only because People on the Internet seem to keep
suggesting it. I believe madness that way lies. I dislike giving up, so for
lack of a better option, perhaps we might need to ditch docker-compose.
I've got a PR open that wraps docker compose in a unified script. It
wouldn't be entirely unreasonable to shift a bit more logic into it.
Another possibility is to see if the infra folks would mind adding python
and docker-compose. I'm not sure adding python to the mix on those boxen is
a good idea, though, even if they're willing.
On Tue, Mar 14, 2017 at 6:03 PM Leif Hedstrom <[email protected]> wrote:
> On Mar 14, 2017, at 6:15 PM, Chris Lemmons <[email protected]> wrote:
>
> Honestly, the key is hosting. If we have a host for CI that runs the basic
> build steps, we can configure any solution to build all the changes on
> branches of a collection of repos on Github. Pretty much all the
reasonable
> options have a status update script on GitHub, which integrates it quite
> nicely. (And therein might lie the rub. I think GitHub ties status updates
> to "push permission", which may be false for everyone on the main repo,
> since it's just a mirror.) But direct integration or no, we'd be able to
go
> look at the results and even download the binary, install it on a test
> system and watch it go.
So, we do not have the Jenkins master have “write permission” into the
Github repo. I asked Infra before, and they said no, but I’ll try again.
However, things can still work reasonably well, since any registered Github
accounts is able to comment on a PR / issue. So, no, we can’t set labels
etc. automatically from the Jenkins master, but we get pretty good feedback
on what happens with the builds. See e.g.
https://github.com/apache/trafficserver/pull/1581 <
https://github.com/apache/trafficserver/pull/1581>
Cheers,
— leif
>
> Now, that doesn't get us automatic builds from first-time or probably even
> very occasional contributors. But stick builds on the most frequent
> contributors' clones and we get 95% of the benefit without solving any of
> the actually hard problems.
>
> We'd need a host, though.
>
> On Tue, Mar 14, 2017 at 5:06 PM Leif Hedstrom <[email protected]> wrote:
>
>>
>>> On Mar 13, 2017, at 8:44 AM, Chris Lemmons <[email protected]> wrote:
>>>
>>> To me, the key features of CI are that a) it builds each branch
>>> automatically, b) notifies affected parties when all is not well, and c)
>>> manages the artefacts in a reasonable way. Additionally, we're a lot
more
>>> useful when we're writing neat software and not spending out time
>> managing
>>> CI, so it should be as automatic as reasonable. We're using github for
>> PRs,
>>> so if it's at all possible to get automatic PR tagging with build
>>> information, that is greatly desirable. Knowing that the PR breaks the
>>> build prior to merging it can save quite a bit of time. :)
>>
>>
>> My $0.25: My experience is that making as much CI build / tests on pull
>> requests, *before* they are landed, gives the most bang for the buck. But
>> that might not work well for you, since you can’t use Github right?
>>
>> — leif
>>
>>>
>>> I've not used BuildBot, but it feels... svney?
>>>
>>> Jenkins can do all of the above, though basically all those features are
>>> plugins. There's a "build per branch" plugin that uses branches to
>>> automatically make builds. There's a variety of notifier plugins. There
>> are
>>> some artefact management plugins. There is a "build PRs" plugin that's
>>> specifically designed for GitHub. Jenkins isn't much on its own, but
>>> properly adorned with plugins, it can do most of what we'd want, I
think.
>>>
>>> I've also had extensive experience with Bamboo, Atlassian's
closed-source
>>> solution in the same suite as Jira. Bamboo is usable gratis for OSS in
>> the
>>> same way we currently use Apache Jira.
>>>
>>> Bamboo also has plugins, but the majority of it's features are good to
go
>>> immediately. There's an automatic checkbox for "build per branch". The
>>> artefacts are managed fairly automatically, especially if you fill in
the
>>> "build expiration" boxes. Notification is automatic and easy to
>> configure.
>>> It's got a (free) plugin for PR statuses.
>>>
>>> In any case, we'll need to manually configure the valid lists of
>>> contributors. For security reasons, we probably can't just run whatever
>> PR
>>> is created without any prior contact. :/ In Jenkins, this looks like
>>> maintaining a "white list" of legal contributors for a PR inside the PR
>>> plugin. In Bamboo, it looks like listing the committer forks as copied
>>> projects. Either way is fairly manageable.
>>>
>>> Jenkins and Bamboo both run on Java. So... no winner there. :)
>>>
>>> I think the major question is one of hosting. No matter what solution we
>>> use, we need a few cycles, a network interface, and a bit of disk space
>> to
>>> run it. The apache jenkins appears to have almost all the stuff we need.
>> It
>>> does not appear to have docker-compose, which we're leveraging fairly
>>> significantly at the moment. docker-compose, however, is Apache
licensed,
>>> so we could theoretically ship it inside the repo. Not sure I like that
>>> option, though. We could also switch off of compose and put the relevant
>>> options into our build script directly. Not sure I like that option,
>> either.
>>>
>>> We'd have the most flexibility if we could get one or more companies to
>>> donate a publicly accessible host (or even theoretically, a build
slave),
>>> assuming that doesn't break Apache's rules. The CI doesn't need a ton of
>>> gas, but the more oomph it has, the more granularly it can build and
more
>>> aggressively we can test.
>>>
>>> On Sun, Mar 12, 2017 at 6:54 PM Eric Friedrich (efriedri) <
>>> [email protected]> wrote:
>>>
>>> Hey All-
>>> I’d played around before with Travis CI for Continuous Integration
>>> builds, but never actually set it up for the public repo.
>>>
>>> I know some others on the list have tried out comparable services. Does
>>> anyone have experience or suggestions to share?
>>>
>>> Also, we can now get access to Apache Buildbot (
>>> https://ci.apache.org/buildbot.html) and an Apache hosted Jenkins (
>>> https://cwiki.apache.org/confluence/display/INFRA/Jenkins)
>>>
>>> I think Traffic Server has their own separate Jenkins server so they can
>>> hit more platforms, but with the latest build changes, pretty much all
we
>>> require is access to a docker daemon
>>>
>>> —Eric
>>
>>
