It seems that there is intermittent Apache-wide degradation of Travis CI services -- I was looking at https://travis-ci.org/apache today and there appeared to be a stretch of 3-4 hours where no queued builds on github.com/apache were running at all. I initially thought that the issue was contention with other Apache projects but even with round-robin allocation and a concurrency limit (e.g. no Apache project having more than 5-6 concurrent builds) that wouldn't explain why NO builds are running.
This is obviously disturbing given how reliant we are on Travis CI to validate patches to be merged. I've opened a support ticket with Travis CI to see if they can provide some insight into what's going on. There is also an INFRA ticket where other projects have reported some similar experiences https://issues.apache.org/jira/browse/INFRA-18533 As a meta-comment, at some point Apache Arrow is going to need to move off of public CI services for patch validation so that we can have unilateral control over scaling our build / test resources as the community grows larger. As the most active merger of patches (I have merged over 50% of pull requests over the project's history) this affects me greatly as I am often monitoring builds on many open PRs so that I can merge them as soon as possible. We are often resorting to builds on contributor's forks (assuming they have enabled Travis CI / Appveyor) As some context around Travis CI in particular, in January Travis CI was acquired by Idera, a private equity (I think?) developer tools conglomerate. It's likely that we're seeing some "maximize profit, minimize costs" behavior in play, so the recent experience could become the new normal. - Wes
