Hi all, I'd like to discuss options to execute ROS tests (per runner) more efficiently, and explore the option of running them on PreCommit, as opposed to PostCommit as they run today.
The SDK provides a set of tests called "RunnableOnService" (aka ROS) that can be applied to a runner and validate it (correctly) supports SDK features. It's 300+ tests in total (batch + streaming) and it clearly takes time, and that is why it runs on PostCommit. I think we should look for a configuration where this is executed more efficiently and if possible run on PreCommit since runners are encouraged to rely on those tests and it's better to know of breaking changes before hand. This came up somewhere in this <https://github.com/apache/incubator-beam/pull/1055> conversation, and the highlights are basically: Kenneth Knowles suggested we might parallelize sub-builds in the following manner: 1. Run unit tests. 2. (sub tasks) Run ROS tests for each runner in parallel, skipping unit tests. I was wondering if we could setup Jenkins to run ROS per runner only of there was a code change for that runner - of course SDK changes will probably have to run ROS for all runners, but that might still be an optimization. I think one of Beam's best sell-points is it's extensive testing framework, and the fact that runners can be validated across capabilities, but it would be best to know of runner-braking changes before merging to master. Thoughts ? Thanks, Amit
