[
https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070049#comment-16070049
]
ASF GitHub Bot commented on FLINK-7047:
---------------------------------------
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/4233
[FLINK-7047] [travis] Reorganize build profiles
Builds on-top of #4226 and #4227.
This PR reorganizes the test groups.
1. core - executes tests for core, runtime, streaming-java, scala, etc
2. libraries - executes tests for modules libraries and storm
3. connectors - executes tests for connectors and wikiedits
4. tests - executes tests for flink-tests and examples
5. misc - executes tests for yarn, mesos, fs-tests, dist
The main change (and worst part) is that every module (or their respective
parent) now have a `run-tests` profile that is activated when the specific
property for that groups was set.
For example, tests for `flink-libraries` are activated if the
`flink.test.lib` property is set.
For this to work it was necessary to **disable the test execution by
default**. Execution of tests is now **strictly opt-in** for each module (or
their parent).
To keep the number of build profiles <= 10 I've also reduced the
combinations of jdk+scala+hadoop that we're testing to 2:
* oraclejdk8 + scala11 + hadoop2.8.0
* openjdk7 + scala10 + hadoop2.4.1
For the jdk7 profile max build times hover around 45 minutes for the core
tests; for jdk8 the same build takes 40 minutes.
I tried to reduce this further my cutting out parts of the compilation,
specifically the scala modules which easily add 6-8 minutes, but i couldn't
find a way that properly works and is maintainable enough. Instead of fiddling
with maven it's probably easier just to split the entire repo and call it a day.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 7047
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4233.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4233
----
commit 94e2b5bf02c2f5e04ecfd7af18f063a773e603c3
Author: zentol <[email protected]>
Date: 2017-06-29T21:17:01Z
[FLINK-7048] [travis] Define javadoc skipping in travis watchdog script
commit 1d1cc7549c262985f7d9b98f079297d40210c73d
Author: zentol <[email protected]>
Date: 2017-06-29T19:31:23Z
[FLINK-7046] [travis] Hide download logging messages
commit e8cb22ff9bac40fe4df69206b390f9d7040ea150
Author: zentol <[email protected]>
Date: 2017-06-29T15:36:21Z
[FLINK-7047] [travis] Reorganize build profiles
----
> Reorganize build profiles
> -------------------------
>
> Key: FLINK-7047
> URL: https://issues.apache.org/jira/browse/FLINK-7047
> Project: Flink
> Issue Type: Improvement
> Components: Tests, Travis
> Affects Versions: 1.4.0
> Reporter: Chesnay Schepler
> Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> With the current build times once again hitting the timeout it is time to
> revisit our approach.
> The current approach of splitting all tests by name, while easy to maintain
> or extend, has the big disadvantage that it's fairly binary in regards to the
> timeout: either we're below the timeout and all builds pass, or we're above
> and the entire merging process stalls. Furthermore, it requires all modules
> to be compiled.
> I propose a different approach by which we bundle several modules, only
> execute the tests of these modules and skip the compilation of some modules
> that are not required for these tests.
> 5 groups are my current suggestion, which will result in 10 build profiles
> total.
> The groups are:
> # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb
> # *libraries* - flink-libraries and flink-storm
> # *connectors* - flink-connectors, flink-connector-wikiedits,
> flink-tweet-inputformat
> # *tests* - flink-tests
> # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples,
> flink-dist
> To not increase the total number of profiles to ridiculous numbers i also
> propose to only test against 2 combinations of jdk+hadoop+scala:
> # oraclejdk8 + hadoop 2.8.0 + scala 2.11
> # openjdk7 + hadoop 2.4.1 + scala 2.10
> My current estimate is that this will cause profiles to take at most 40
> minutes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
