+1 to making these cheaper. Another idea I've had, though I'm not sure how hard it would be to implement, is that many ROS tests are trivial pipelines and dominated by the overhead of invoking the service itself. I wonder how hard it would be to transparently "pack" many pipelines into single job requests whose success would be a very strong (though not perfect) indicator that each pipeline would have succeeded individually. Of course not all tests are structured in this manner, but from my experience most are, and this could be significant savings.
On Thu, Oct 20, 2016 at 12:24 PM, Amit Sela <[email protected]> wrote: > Hi all, > > I'd like to discuss options to execute ROS tests (per runner) more > efficiently, and explore the option of running them on PreCommit, as > opposed to PostCommit as they run today. > > The SDK provides a set of tests called "RunnableOnService" (aka ROS) that > can be applied to a runner and validate it (correctly) supports SDK > features. > It's 300+ tests in total (batch + streaming) and it clearly takes time, and > that is why it runs on PostCommit. > I think we should look for a configuration where this is executed > more efficiently and if possible run on PreCommit since runners are > encouraged to rely on those tests and it's better to know of breaking > changes before hand. > > This came up somewhere in this > <https://github.com/apache/incubator-beam/pull/1055> conversation, and the > highlights are basically: > > Kenneth Knowles suggested we might parallelize sub-builds in the following > manner: > > 1. Run unit tests. > 2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit > tests. > > I was wondering if we could setup Jenkins to run ROS per runner only of > there was a code change for that runner - of course SDK changes will > probably have to run ROS for all runners, but that might still be an > optimization. > > I think one of Beam's best sell-points is it's extensive testing framework, > and the fact that runners can be validated across capabilities, but it > would be best to know of runner-braking changes before merging to master. > > Thoughts ? > > Thanks, > Amit
