[jira] [Commented] (FLINK-8036) Consider using gradle to build Flink
[ https://issues.apache.org/jira/browse/FLINK-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261371#comment-16261371 ] Greg Hogan commented on FLINK-8036: --- Are many developers running tests from the command line or using an IDE for incremental compilation and selectively running tests? > Consider using gradle to build Flink > > > Key: FLINK-8036 > URL: https://issues.apache.org/jira/browse/FLINK-8036 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu > > Here is summary from Lukasz over this thread > (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) > w.r.t. performance boost from using gradle: > Maven performs parallelization at the module level, an entire module needs > to complete before any dependent modules can start, this means running all > the checks like findbugs, checkstyle, tests need to finish. Gradle has task > level parallelism between subprojects which means that as soon as the > compile and shade steps are done for a project, and dependent subprojects > can typically start. This means that we get increased parallelism due to > not needing to wait for findbugs, checkstyle, tests to run. I typically see > ~20 tasks (at peak) running on my desktop in parallel. > Flink should consider using gradle - on Linux with SSD, a clean build takes > an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8036) Consider using gradle to build Flink
[ https://issues.apache.org/jira/browse/FLINK-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261193#comment-16261193 ] Ted Yu commented on FLINK-8036: --- Last note: when I see gradlew run selected test(s) so fast, I wish maven doesn't have to go over all the modules before coming to the module where the test resides. > Consider using gradle to build Flink > > > Key: FLINK-8036 > URL: https://issues.apache.org/jira/browse/FLINK-8036 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu > > Here is summary from Lukasz over this thread > (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) > w.r.t. performance boost from using gradle: > Maven performs parallelization at the module level, an entire module needs > to complete before any dependent modules can start, this means running all > the checks like findbugs, checkstyle, tests need to finish. Gradle has task > level parallelism between subprojects which means that as soon as the > compile and shade steps are done for a project, and dependent subprojects > can typically start. This means that we get increased parallelism due to > not needing to wait for findbugs, checkstyle, tests to run. I typically see > ~20 tasks (at peak) running on my desktop in parallel. > Flink should consider using gradle - on Linux with SSD, a clean build takes > an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8036) Consider using gradle to build Flink
[ https://issues.apache.org/jira/browse/FLINK-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246715#comment-16246715 ] Chesnay Schepler commented on FLINK-8036: - I can see the benefit, but as it stands I'm against it. As Fabian said, this would entail changing many integral parts of the system, not to mention how users build custom flink versions. Furthermore, there are just really trivial workarounds for the described use-case: Step1: compile Flink (without checkstyle etc.) Step2: invoke checkstyle, findbugs, tests etc. for each submodule in a separate process This is rather trivial to implement and should buy you plenty of time. > Consider using gradle to build Flink > > > Key: FLINK-8036 > URL: https://issues.apache.org/jira/browse/FLINK-8036 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu > > Here is summary from Lukasz over this thread > (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) > w.r.t. performance boost from using gradle: > Maven performs parallelization at the module level, an entire module needs > to complete before any dependent modules can start, this means running all > the checks like findbugs, checkstyle, tests need to finish. Gradle has task > level parallelism between subprojects which means that as soon as the > compile and shade steps are done for a project, and dependent subprojects > can typically start. This means that we get increased parallelism due to > not needing to wait for findbugs, checkstyle, tests to run. I typically see > ~20 tasks (at peak) running on my desktop in parallel. > Flink should consider using gradle - on Linux with SSD, a clean build takes > an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8036) Consider using gradle to build Flink
[ https://issues.apache.org/jira/browse/FLINK-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245390#comment-16245390 ] Fabian Hueske commented on FLINK-8036: -- I think this will be very hard. Flink's build management has grown a lot over time and is far from trivial with the dependency, code style, header check, artifact generation, and especially shading configuration. Re-implementing all of this in Gradle (or any other build system) would be a large effort. However, IMO this might not even be the biggest issue. The Flink community (or at least a good number of people) knows Maven quite well by now. All that knowledge would be lost and would have to be regained for a different tool if we would switching. The community would depend quite heavily on whoever would port the build until we gained enough experience with Gradle or whatever. > Consider using gradle to build Flink > > > Key: FLINK-8036 > URL: https://issues.apache.org/jira/browse/FLINK-8036 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu > > Here is summary from Lukasz over this thread > (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) > w.r.t. performance boost from using gradle: > Maven performs parallelization at the module level, an entire module needs > to complete before any dependent modules can start, this means running all > the checks like findbugs, checkstyle, tests need to finish. Gradle has task > level parallelism between subprojects which means that as soon as the > compile and shade steps are done for a project, and dependent subprojects > can typically start. This means that we get increased parallelism due to > not needing to wait for findbugs, checkstyle, tests to run. I typically see > ~20 tasks (at peak) running on my desktop in parallel. > Flink should consider using gradle - on Linux with SSD, a clean build takes > an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8036) Consider using gradle to build Flink
[ https://issues.apache.org/jira/browse/FLINK-8036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245197#comment-16245197 ] Bowen Li commented on FLINK-8036: - it would be great! > Consider using gradle to build Flink > > > Key: FLINK-8036 > URL: https://issues.apache.org/jira/browse/FLINK-8036 > Project: Flink > Issue Type: Improvement >Reporter: Ted Yu > > Here is summary from Lukasz over this thread > (http://search-hadoop.com/m/Beam/gfKHFVh4NM151XIu1?subj=Re+DISCUSS+Move+away+from+Apache+Maven+as+build+tool) > w.r.t. performance boost from using gradle: > Maven performs parallelization at the module level, an entire module needs > to complete before any dependent modules can start, this means running all > the checks like findbugs, checkstyle, tests need to finish. Gradle has task > level parallelism between subprojects which means that as soon as the > compile and shade steps are done for a project, and dependent subprojects > can typically start. This means that we get increased parallelism due to > not needing to wait for findbugs, checkstyle, tests to run. I typically see > ~20 tasks (at peak) running on my desktop in parallel. > Flink should consider using gradle - on Linux with SSD, a clean build takes > an hour. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
