Chesnay Schepler created FLINK-8745:
---------------------------------------
Summary: Reduce travis usage
Key: FLINK-8745
URL: https://issues.apache.org/jira/browse/FLINK-8745
Project: Flink
Issue Type: Improvement
Components: Build System, Travis
Affects Versions: 1.4.0, 1.5.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
We've been notified by INFRA that our travis usage is exceedingly high.
There are various things we could look into short- and long term:
h2. Short-term
h3. Reduce number of jobs
We currently run 12 job for each pr/push.
The first 10 jobs belong to 2 groups, with each group representing one test run
of Flink against a specific hadoop version.
Given that the majority of changes made to Flink do not impact our
compatibility with hadoop we could drop one of the groups and instead rely on
daily cron jobs. This alone would cut our travis usage by 40%.
Once the migration to flip6 is done we can drop the remaining 2 jobs,
increasing the reduction to 60%.
h3. Reduce number of builds
Travis is run for every PR, regardless of what change was made, even if it was
something trivial as removing a trailing space in a documentation file. From
time to time it also happens that new commits are pushed in a PR solely to
trigger a new build to get that perfect green build.
Instead we could look into manually triggering travis for pull requests, that
is with a bot.
h2. Long-term
h3. Incremental builds
Making the build dependent on the changes made has been brought up a few times
now. This would in particular benefit cases where connectors/libraries are
modified as they generally have few dependents. We would still have to run
everything though if changes are made to the core modules.
h3. Repository split
The most painful of them all, but in my opinion also the most promising. With
separate repositories for the core flink modules (flink-runtime etc),
flink-connectors and flink-libraries would cut downright skip the compilation
for a large number of modules.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
