[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4233 I think this looks good. It's basically a change now restricted to the build system only. +1 to give it a try on master and see how things go with it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 yes we could build with parallel builds, although i encountered the storm module acting up once. It also, at least in early prototypes, didn't provide a big improvement. (iirc 1 minute when building Flink completely) The results may be different now. That said, I would defer further tinkering on small details to follow-ups, since local builds are still timing out and I don't want to delay this further unless something is actually broken. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/4233 @aljoscha true. Now that proportionally more time is spent compiling we may be able to enable parallel builds for the compilation phase. I think most issues were with running tests where dependencies were not properly configured. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4233 I would hope so, because the number of version combinations we check is also halved. ð --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/4233 +1; total run time has dropped by almost half --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 I've reworked the "misc" group. There is no longer a dedicated list of modules for this group, but instead we test all modules that aren't included in any other group. All other groups have a comma-separated list of modules to test, like `flink-runtime,flink-core`. For the "Misc" group we concatenate those lists and negate them, like `!flink-runtime,!flink-core`. What's left is the modules that aren't contained in another group, which means all new modules would be run automatically. The compilation step of this group simply builds flink completely; this was basically required anyway since the group contained flink-dist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 I've got an idea on how to make sure that all modules are guaranteed to be covered, sit tight... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 There's no verification, and i don't think that this is a blocking issue. We can verify it now manually and open an issue to add some magic later on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4233 This looks nice now! Is there verification, though, that we didn't forget any modules in the list of tested modules? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @aljoscha @rmetzger I've reworked this PR, could you take another look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 We can do this in a more centralized way using the `-pl` option as @aljoscha suggested (see https://github.com/zentol/flink/tree/7047_alt2), but this is blocked by FLINK-7112 & FLINK-7112. https://travis-ci.org/zentol/flink/builds/250679622 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 Currently experimenting with specifying every module to test using the `-pl` option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` now works as it should, execution tests and stuff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger I can fix the issue with verify by reverting the profile logic. Instead of activating the surefire plugin in each module, the plugin is enabled by default in the root pom but disabled in the profile of each module if the property is set --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger Now i understand why the tests aren't run. I thought that I was activating the profiles with the properties in the root pom.xml, but this isn't the case. I was just setting the system properties in the travis scripts... back to the drawing board i guess... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 WIth the existing surefire configuration we also cannot do this in 2 steps, one for compilation and one for tests. It is not possible to only run tests with any maven invocation that includes a life-cycle (i.e verify), which means that you have to recompile some modules twice which isn't an option because scala compilation is so utterly horrible. You _could_ theoretically cd into each project after compilation and directly invoke the surefire plugin, but this doesn't work as we have 2 executions (1 for unit tests, 1 for IT cases), and when invoking a plugin directly you can at most run 1 execution and it has to be named either "default-test" or "default-cli". This would be possible in maven 3.3.1, but we can't use that without messing with the shading, We could use maven 3.3.X if we split the repo and add a separate one for flink-dist, but that's not gonna happen anytime soon. We could directly invoke the surefire plugin if we merged our surefire executions, but then we're still left with the problem of having to keep the travis scripts in sync with the directory structure of the project. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 another more sane option would be to add a new module for each group that has a dependency for each module we want tested. THEN we could use the `-am` option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 corrected my previous comment regarding the problems of `-am` and `-amd`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` _should_ run all tests since the default for all added properties is true. But yes, it currently doesn't. I don't quite understand this as the default for all groups is true (i.e. all test profiles should activate) and I'm not modifying the phases of these profiles. Maven's `-am` and `-amd` options don't work for us since these don't build dependencies marked as provided, i.e. most core flink dependencies. You can easily verify this yourself. Maven's `-pl` command is a bit brittle in general as we have to include every single module; we can't just point to flink-libraries and call it a day. This adds a maintainability problem; a new module will not be tested on travis if the contributor/reviewer isn't aware of the magic in the travis scripts. This is less likely with the approach in this PR as part of the magic is in the poms which are more likely to be seen by a developer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4233
@aljoscha this may be caused by publishing pom's with unresolved
`${scala.binary.version}`. I am planning to file a ticket after #4221 is
merged. The StackOverflow link looks promising.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4233 There's also this interesting approach which limits changes to the root POM: https://stackoverflow.com/a/39562385/4770039 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4233
I think we should be able to do this as a two-step process without changing
the poms:
```
mvn install -pl flink-tests,flink-foo-bar -am -DskipTests
-Dmaven.javadoc.skip=true
mvn verify -pl flink-tests,flink-foo-bar -Dmaven.javadoc.skip=true
```
The first line will make sure that we have all the dependencies built, the
second one only executes the tests in the given submodules. We cannot do it as
one step because running `mvn verify -pl flink-tests,... -am` would run the
tests also on all dependencies of `flink-test,...`.
However, it seems we can't do this because the second incantation fails
with
```
[WARNING] Some problems were encountered while building the effective model
for org.apache.flink:flink-scala-shell_2.11:jar:1.4-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.flink:flink-scala-shell_${scala.binary.version}:[unknown-version],
/Users/aljoscha/Dev/flink/flink-scala-shell/pom.xml, line 30, column 14
```
that is including the Scala version in our artifact IDs broke this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/4233 So executing `mvn clean verify` won't run the tests, right? Have you considered doing this, basically without touching the pom files at all? Basically by having a script that says "cd flink-core; mvn install cd flink-runtime; mvn install cd flink-tests; mvn verify " I think there's also a command to tell maven "build all your dependencies" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 I've moved a few more modules into the misc. group: * flink-metrics * flink-java8 * flink-streaming-contrib * flink-scala-shell For the core group we now only compile modules that the core modules actually depend on, reducing the core build time to 30 minutes: https://travis-ci.org/zentol/flink/builds/249710980 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 Note that we can further reduce the build-time of the core profile to ~35 minutes by skipping the compilation libraries/connectors. At least we should, if it weren't for flink-7070... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 I would merge this PR in any case. It's a nice short-term option that is more stable than the previous approach; regardless of how many times we add in flink-runtime we can still test connectors/libs and vice versa. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/4233 @zentol could you start a mailing list discussion and describe your preference for splitting the tests in this manner or splitting the repo? Despite heroic efforts by you and Robert keeping the test times under the timeout continues to be a Sisyphean task. The number of tests never truly drops so the test times will only decrease on the off chance that TravisCI bumps the number of cores (or Google increases CPU performance, which they have never done). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
