Stephan Ewen commented on FLINK-8745:

Staged builds sound promising.

I am somewhat unsure about moving other heavy tests (Kafka connector, Yarn 
integration, etc) to something like cron jobs. I fear that occasional failures 
of cron job tests would not be treated the same way as failures in the master 

The fact that some of these heavy tests run often has helped in the past to 
uncover corner case instabilities/races/bugs.

> Reduce travis usage
> -------------------
>                 Key: FLINK-8745
>                 URL: https://issues.apache.org/jira/browse/FLINK-8745
>             Project: Flink
>          Issue Type: Improvement
>          Components: Build System, Travis
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Critical
> We've been notified by INFRA that our travis usage is exceedingly high.
> There are various things we could look into short- and long term:
> h2. Short-term
> h3. Reduce number of jobs
> We currently run 12 job for each pr/push.
> The first 10 jobs belong to 2 groups, with each group representing one test 
> run of Flink against a specific hadoop version.
> Given that the majority of changes made to Flink do not impact our 
> compatibility with hadoop we could drop one of the groups and instead rely on 
> daily cron jobs. This alone would cut our travis usage by 40%.
> Once the migration to flip6 is done we can drop the remaining 2 jobs, 
> increasing the reduction to 60%.
> h3. Reduce number of builds
> Travis is run for every PR, regardless of what change was made, even if it 
> was something trivial as removing a trailing space in a documentation file. 
> From time to time it also happens that new commits are pushed in a PR solely 
> to trigger a new build to get that perfect green build.
> Instead we could look into manually triggering travis for pull requests, that 
> is with a bot.
> h2. Long-term
> h3. Incremental builds
> Making the build dependent on the changes made has been brought up a few 
> times now. This would in particular benefit cases where connectors/libraries 
> are modified as they generally have few dependents. We would still have to 
> run everything though if changes are made to the core modules.
> h3. Repository split
> The most painful of them all, but in my opinion also the most promising. With 
> separate repositories for the core flink modules (flink-runtime etc), 
> flink-connectors and flink-libraries would cut downright skip the compilation 
> for a large number of modules.

This message was sent by Atlassian JIRA

Reply via email to