I’ve got a Jenkins project setup on ASF Jenkins: https://builds.apache.org/view/Incubator%20Projects/job/incubator-trafficcontrol-master-build/
No docker-compose, so I’ll wait for Chris’s PR before activating this for real. On the plus side, this ASF Jenkins server comes pre-authenticated to Github with the asfgit (admin) credentials. I think this means we can use the PR builder with build status updates —Eric On Mar 15, 2017, at 4:11 PM, Chris Lemmons <[email protected]<mailto:[email protected]>> wrote: So, after some investigation, I've circled back on the mounting docker-compose sideways and letting it manage sibling containers. It appears that docker folks have already tamed the maddest of the madness. There's a relatively reasonably supported script for doing it that I was able to reasonably incorporate in the packaging script I had already put together. I've updated the PR to include this: https://github.com/apache/incubator-trafficcontrol/pull/347 If you have docker-compose, it will use it. If you don't, it will run it inside a container. Caveats apply, but none that are likely in practice, I think. This reduces the requirements to: - git - bash - docker All of which are satisfied by the ubuntu hosts on the ASF build infrastructure. On Wed, Mar 15, 2017 at 8:57 AM Jeff Elsloo <[email protected]> wrote: Docker isn't required to build the software, it's just another option. There's a build script, `build/build.sh`, that works just fine so long as you have the dependencies required to successfully build all components. I only mention this because if Docker is going to gate our ability to perform CI out in the open, we still have the `build.sh` option. I was able to use the build script to successfully build all components from master yesterday. -- Thanks, Jeff On Tue, Mar 14, 2017 at 8:27 PM, Chris Lemmons <[email protected]> wrote: Yeah, there're unfortunately good reasons not to have any accounts with write permission in the GitHub repo. It can cause all sorts of problems if anything were actually pushed. It also allows lots of other things like editing other people's comments. GitHub should really separate that out for the purpose of bots with minimum required access anyway. But yeah, without write, it's a no-go. Comments are a very reasonable alternative, though. It's definitely worth a few minutes to get things set up on the ASF Jenkins if those ubuntu slaves have the requirements. As it stands, the only requirements for a build are: - git - bash - docker - docker-compose - which requires python I believe the first three are satisfied by the build hosts. I don't know that docker-compose is available. It's 100% worth finding out, though. If it's not, we can do one of: - Run docker-compose inside a docker instance with the docker port forwarded into the container to allow it to manage sibling containers. - Re-create the subset of docker-compose behaviour that we actually use in a build script. - Give up. I mention the first option only because People on the Internet seem to keep suggesting it. I believe madness that way lies. I dislike giving up, so for lack of a better option, perhaps we might need to ditch docker-compose. I've got a PR open that wraps docker compose in a unified script. It wouldn't be entirely unreasonable to shift a bit more logic into it. Another possibility is to see if the infra folks would mind adding python and docker-compose. I'm not sure adding python to the mix on those boxen is a good idea, though, even if they're willing. On Tue, Mar 14, 2017 at 6:03 PM Leif Hedstrom <[email protected]> wrote: On Mar 14, 2017, at 6:15 PM, Chris Lemmons <[email protected]> wrote: Honestly, the key is hosting. If we have a host for CI that runs the basic build steps, we can configure any solution to build all the changes on branches of a collection of repos on Github. Pretty much all the reasonable options have a status update script on GitHub, which integrates it quite nicely. (And therein might lie the rub. I think GitHub ties status updates to "push permission", which may be false for everyone on the main repo, since it's just a mirror.) But direct integration or no, we'd be able to go look at the results and even download the binary, install it on a test system and watch it go. So, we do not have the Jenkins master have “write permission” into the Github repo. I asked Infra before, and they said no, but I’ll try again. However, things can still work reasonably well, since any registered Github accounts is able to comment on a PR / issue. So, no, we can’t set labels etc. automatically from the Jenkins master, but we get pretty good feedback on what happens with the builds. See e.g. https://github.com/apache/trafficserver/pull/1581 < https://github.com/apache/trafficserver/pull/1581> Cheers, — leif Now, that doesn't get us automatic builds from first-time or probably even very occasional contributors. But stick builds on the most frequent contributors' clones and we get 95% of the benefit without solving any of the actually hard problems. We'd need a host, though. On Tue, Mar 14, 2017 at 5:06 PM Leif Hedstrom <[email protected]> wrote: On Mar 13, 2017, at 8:44 AM, Chris Lemmons <[email protected]> wrote: To me, the key features of CI are that a) it builds each branch automatically, b) notifies affected parties when all is not well, and c) manages the artefacts in a reasonable way. Additionally, we're a lot more useful when we're writing neat software and not spending out time managing CI, so it should be as automatic as reasonable. We're using github for PRs, so if it's at all possible to get automatic PR tagging with build information, that is greatly desirable. Knowing that the PR breaks the build prior to merging it can save quite a bit of time. :) My $0.25: My experience is that making as much CI build / tests on pull requests, *before* they are landed, gives the most bang for the buck. But that might not work well for you, since you can’t use Github right? — leif I've not used BuildBot, but it feels... svney? Jenkins can do all of the above, though basically all those features are plugins. There's a "build per branch" plugin that uses branches to automatically make builds. There's a variety of notifier plugins. There are some artefact management plugins. There is a "build PRs" plugin that's specifically designed for GitHub. Jenkins isn't much on its own, but properly adorned with plugins, it can do most of what we'd want, I think. I've also had extensive experience with Bamboo, Atlassian's closed-source solution in the same suite as Jira. Bamboo is usable gratis for OSS in the same way we currently use Apache Jira. Bamboo also has plugins, but the majority of it's features are good to go immediately. There's an automatic checkbox for "build per branch". The artefacts are managed fairly automatically, especially if you fill in the "build expiration" boxes. Notification is automatic and easy to configure. It's got a (free) plugin for PR statuses. In any case, we'll need to manually configure the valid lists of contributors. For security reasons, we probably can't just run whatever PR is created without any prior contact. :/ In Jenkins, this looks like maintaining a "white list" of legal contributors for a PR inside the PR plugin. In Bamboo, it looks like listing the committer forks as copied projects. Either way is fairly manageable. Jenkins and Bamboo both run on Java. So... no winner there. :) I think the major question is one of hosting. No matter what solution we use, we need a few cycles, a network interface, and a bit of disk space to run it. The apache jenkins appears to have almost all the stuff we need. It does not appear to have docker-compose, which we're leveraging fairly significantly at the moment. docker-compose, however, is Apache licensed, so we could theoretically ship it inside the repo. Not sure I like that option, though. We could also switch off of compose and put the relevant options into our build script directly. Not sure I like that option, either. We'd have the most flexibility if we could get one or more companies to donate a publicly accessible host (or even theoretically, a build slave), assuming that doesn't break Apache's rules. The CI doesn't need a ton of gas, but the more oomph it has, the more granularly it can build and more aggressively we can test. On Sun, Mar 12, 2017 at 6:54 PM Eric Friedrich (efriedri) < [email protected]> wrote: Hey All- I’d played around before with Travis CI for Continuous Integration builds, but never actually set it up for the public repo. I know some others on the list have tried out comparable services. Does anyone have experience or suggestions to share? Also, we can now get access to Apache Buildbot ( https://ci.apache.org/buildbot.html) and an Apache hosted Jenkins ( https://cwiki.apache.org/confluence/display/INFRA/Jenkins) I think Traffic Server has their own separate Jenkins server so they can hit more platforms, but with the latest build changes, pretty much all we require is access to a docker daemon —Eric
