Re: Gradle Status [April 11]

[Nathan Fisher]

Great thanks for that! Makes perfect sense.

On Wed, Apr 11, 2018 at 7:59 PM, Kenneth Knowles  wrote:

> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately
> it was a community-driven decision: more people (as in "more than just me"
> :-) jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
> wrote:
>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits
>> were considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner 
>>> wrote:
>>>
 Thanks everyone for the continued effort towards the Gradle migration.
 As a high-level summary of our progress since Friday: we have a viable
 build, with a number of minor issues that we're still working out. Please
 take a look at the new documentation in our contribution guide and log any
 bugs that you find.

 Here's a more detailed view of improvements from just the past few
 days..

 Release artifacts:
 *  Pom.xml generation logic now in master [1]
 * Nightly snapshots are now produced using Gradle [2]
 * Excluded modules propagated to dependencies when generating * pom.xml
 * Artifact JARs are properly shaded [3]
 * Working on fixing dependency scopes in generated pom [4]
 PreCommits / Postcommits:
 * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
 [7] [8] [9]
 * Jenkins results now include JUnit test results [10] and build scan
 for easier debugging [11]
 * Spark ValidatesRunner PostCommit passes [12] [13]
 * Flink ValidatesRunner PostCommit more reliable [14]
 Documentation / IDE Setup:
 * Contribution Guide [15] is now updated with Gradle commands [16] [17]
 Performance Benchmarks:
 * Working on getting Nexmark benchmarks migrated [18]

 If I missed anything, please add it to this thread.

 We are continuing to use this general roadmap:
 (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
 (b) Postcommits migrated to Gradle
 (c) Migrate documentation from maven to Gradle
 (d) Migrate perfkit suites to use Gradle

 Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
 created a separate issue to track post-migration cleanup items:
 BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.


 [1] https://github.com/apache/beam/pull/5054
 [2] https://github.com/apache/beam/pull/5057
 [3] https://github.com/apache/beam/pull/5087
 [4] https://github.com/apache/beam/pull/5098
 [5] https://github.com/apache/beam/pull/5047
 [6] https://github.com/apache/beam/pull/5088
 [7] https://github.com/apache/beam/pull/5086
 [8] https://github.com/apache/beam/pull/5066
 [9] https://github.com/apache/beam/pull/5059
 [10] https://github.com/apache/beam/pull/5045
 [11] https://github.com/apache/beam/pull/5091
 [12] https://github.com/apache/beam/pull/5093
 [13] https://github.com/apache/beam/pull/5069
 [14] https://github.com/apache/beam/pull/5068
 [15] https://beam.apache.org/contribute/contribution-guide/
 [16] https://github.com/apache/beam-site/pull/412
 [17] https://github.com/apache/beam-site/pull/414
 [18] https://github.com/apache/beam/pull/5051
 [19] https://issues.apache.org/jira/browse/BEAM-3249
 [20] https://issues.apache.org/jira/browse/BEAM-4045

 On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for 

Re: Gradle Status [April 11]

[Romain Manni-Bucau]

just created one https://issues.apache.org/jira/browse/BEAM-4057

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-12 18:15 GMT+02:00 Ahmet Altay :
>> Found another blocker in current artifacts creations: there is not
> pom.xml and pom.properties in META-INF. This is used by tools +
> libraries + integrations so it is quite important to not break it
>
> Romain, is there a JIRA for this issues? If not could you create one please?
>
> On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot 
> wrote:
>>
>> Nice !
>> thanks Kenn
>>
>> Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
>>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> *  Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items: BEAM-4045
>> [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle 

Re: Gradle Status [April 11]

[Ahmet Altay]

> Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it

Romain, is there a JIRA for this issues? If not could you create one please?

On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot 
wrote:

> Nice !
> thanks Kenn
>
> Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
>
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>
> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> *  Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit 

Re: Gradle Status [April 11]

[Etienne Chauchot]

Nice !
thanks Kenn
Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner + Netty 
> has been introduced since yesterday.
> Etienne mentioned he has worked toward setting up periodic runs on all 
> runners, so this should help get us towards
> that. We'll probably prefer to build standalone fat jars for selected runners 
> and use those, which is pending unknown
> issues in the shadow config leaving out required dependencies.
> 
> Kenn
> 
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
> > Thanks everyone for the continued effort towards the Gradle migration. As a 
> > high-level summary of our progress since
> > Friday: we have a viable build, with a number of minor issues that we're 
> > still working out. Please take a look at
> > the new documentation in our contribution guide and log any bugs that you 
> > find.
> > 
> > Here's a more detailed view of improvements from just the past few days..
> > 
> > Release artifacts:
> > *  Pom.xml generation logic now in master [1]
> > * Nightly snapshots are now produced using Gradle [2]
> > * Excluded modules propagated to dependencies when generating * pom.xml
> > * Artifact JARs are properly shaded [3]
> > * Working on fixing dependency scopes in generated pom [4]
> > PreCommits / Postcommits:
> > * All PreCommits and PostCommits migrated [5]; working on deflaking [6] [7] 
> > [8] [9]
> > * Jenkins results now include JUnit test results [10] and build scan for 
> > easier debugging [11]
> > * Spark ValidatesRunner PostCommit passes [12] [13]
> > * Flink ValidatesRunner PostCommit more reliable [14] 
> > Documentation / IDE Setup:
> > * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> > Performance Benchmarks:
> > * Working on getting Nexmark benchmarks migrated [18]
> > 
> > If I missed anything, please add it to this thread.
> > 
> > We are continuing to use this general roadmap:
> > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > (b) Postcommits migrated to Gradle
> > (c) Migrate documentation from maven to Gradle
> > (d) Migrate perfkit suites to use Gradle
> > 
> > Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has created 
> > a separate issue to track post-migration 
> > cleanup items: BEAM-4045 [20]. Feel free to grab any unassigned items off 
> > of either list.
> > 
> > 
> > [1] https://github.com/apache/beam/pull/5054 
> > [2] https://github.com/apache/beam/pull/5057 
> > [3] https://github.com/apache/beam/pull/5087 
> > [4] https://github.com/apache/beam/pull/5098 
> > [5] https://github.com/apache/beam/pull/5047 
> > [6] https://github.com/apache/beam/pull/5088
> > [7] https://github.com/apache/beam/pull/5086 
> > [8] https://github.com/apache/beam/pull/5066 
> > [9] https://github.com/apache/beam/pull/5059 
> > [10] https://github.com/apache/beam/pull/5045 
> > [11] https://github.com/apache/beam/pull/5091 
> > [12] https://github.com/apache/beam/pull/5093 
> > [13] https://github.com/apache/beam/pull/5069 
> > [14] https://github.com/apache/beam/pull/5068 
> > [15] https://beam.apache.org/contribute/contribution-guide/ 
> > [16] https://github.com/apache/beam-site/pull/412 
> > [17] https://github.com/apache/beam-site/pull/414 
> > [18] https://github.com/apache/beam/pull/5051 
> > [19] https://issues.apache.org/jira/browse/BEAM-3249 
> > [20] https://issues.apache.org/jira/browse/BEAM-4045 
> > 
> > On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
> > > I wanted to start a thread to summarize the current state of Gradle 
> > > migration. We've made lots of good progress so
> > > far this week. Here's the status from what I can tell-- please add or 
> > > correct anything I missed:
> > > 
> > > * Release artifacts can be built and published for Snapshot and officlal 
> > > releases [1]
> > > * Gradle-generated releases have been validated with the the Apache Beam 
> > > archetype generation quickstart; still
> > > needs additional validation.
> > > * Generated release pom files have correct project metadata [2]
> > > * The python pre-commits are now working in Gradle [3]
> > > * Ismaël has started a collaborative doc of Gradle tips [4] as we all 
> > > learn the new system-- please add your own.
> > > This will eventually feed into official documentation on the website.
> > > * Łukasz Gajowy is working on migrating performance testing framework [5]
> > > * Daniel is working on updating documentation to refer to Gradle instead 
> > > of maven
> > > 
> > > If I missed anything, please add it to this thread.
> > > 
> > > The general roadmap we're working towards is:
> > > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > > (b) Postcommits migrated to Gradle
> > > (c) Migrate documentation from maven to Gradle
> > > (d) Migrate perfkit suites to use Gradle
> > > 
> > > For those of you that are hacking: thanks for your 

Re: Gradle Status [April 11]

[Romain Manni-Bucau]

Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-12 0:59 GMT+02:00 Kenneth Knowles :
> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately it
> was a community-driven decision: more people (as in "more than just me" :-)
> jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
> wrote:
>>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
>> considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:

 Thanks everyone for the continued effort towards the Gradle migration.
 As a high-level summary of our progress since Friday: we have a viable
 build, with a number of minor issues that we're still working out. Please
 take a look at the new documentation in our contribution guide and log any
 bugs that you find.

 Here's a more detailed view of improvements from just the past few
 days..

 Release artifacts:
 *  Pom.xml generation logic now in master [1]
 * Nightly snapshots are now produced using Gradle [2]
 * Excluded modules propagated to dependencies when generating * pom.xml
 * Artifact JARs are properly shaded [3]
 * Working on fixing dependency scopes in generated pom [4]
 PreCommits / Postcommits:
 * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
 [7] [8] [9]
 * Jenkins results now include JUnit test results [10] and build scan for
 easier debugging [11]
 * Spark ValidatesRunner PostCommit passes [12] [13]
 * Flink ValidatesRunner PostCommit more reliable [14]
 Documentation / IDE Setup:
 * Contribution Guide [15] is now updated with Gradle commands [16] [17]
 Performance Benchmarks:
 * Working on getting Nexmark benchmarks migrated [18]

 If I missed anything, please add it to this thread.

 We are continuing to use this general roadmap:
 (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
 (b) Postcommits migrated to Gradle
 (c) Migrate documentation from maven to Gradle
 (d) Migrate perfkit suites to use Gradle

 Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
 created a separate issue to track post-migration cleanup items: BEAM-4045
 [20]. Feel free to grab any unassigned items off of either list.


 [1] https://github.com/apache/beam/pull/5054
 [2] https://github.com/apache/beam/pull/5057
 [3] https://github.com/apache/beam/pull/5087
 [4] https://github.com/apache/beam/pull/5098
 [5] https://github.com/apache/beam/pull/5047
 [6] https://github.com/apache/beam/pull/5088
 [7] https://github.com/apache/beam/pull/5086
 [8] https://github.com/apache/beam/pull/5066
 [9] https://github.com/apache/beam/pull/5059
 [10] https://github.com/apache/beam/pull/5045
 [11] https://github.com/apache/beam/pull/5091
 [12] https://github.com/apache/beam/pull/5093
 [13] https://github.com/apache/beam/pull/5069
 [14] https://github.com/apache/beam/pull/5068
 [15] https://beam.apache.org/contribute/contribution-guide/
 [16] https://github.com/apache/beam-site/pull/412
 [17] https://github.com/apache/beam-site/pull/414
 [18] https://github.com/apache/beam/pull/5051
 [19] https://issues.apache.org/jira/browse/BEAM-3249
 [20] https://issues.apache.org/jira/browse/BEAM-4045

 On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
> I wanted to start a thread to summarize the current state of Gradle
> 

Re: Gradle Status [April 11]

[Kenneth Knowles]

There are plenty of technical hurdles since Bazel is best suited to an
isolated monorepo. We could probably have overcome those problems with
effort. I got through a few of them in the short sprint I did. Ultimately
it was a community-driven decision: more people (as in "more than just me"
:-) jumped on board and got excited about building out the Gradle build.

Kenn

On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
wrote:

> Is there a document or similar outlining the decision behind Gradle over
> Bazel?
>
> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
> considered when evaluating the two or if it was more a matter of
> community/contributor familiarity.
>
> I found this thread which seems to imply issues around dependency
> management;
>
>
> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>
> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>>
>>> Thanks everyone for the continued effort towards the Gradle migration.
>>> As a high-level summary of our progress since Friday: we have a viable
>>> build, with a number of minor issues that we're still working out. Please
>>> take a look at the new documentation in our contribution guide and log any
>>> bugs that you find.
>>>
>>> Here's a more detailed view of improvements from just the past few days..
>>>
>>> Release artifacts:
>>> *  Pom.xml generation logic now in master [1]
>>> * Nightly snapshots are now produced using Gradle [2]
>>> * Excluded modules propagated to dependencies when generating * pom.xml
>>> * Artifact JARs are properly shaded [3]
>>> * Working on fixing dependency scopes in generated pom [4]
>>> PreCommits / Postcommits:
>>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>>> [7] [8] [9]
>>> * Jenkins results now include JUnit test results [10] and build scan for
>>> easier debugging [11]
>>> * Spark ValidatesRunner PostCommit passes [12] [13]
>>> * Flink ValidatesRunner PostCommit more reliable [14]
>>> Documentation / IDE Setup:
>>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>>> Performance Benchmarks:
>>> * Working on getting Nexmark benchmarks migrated [18]
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> We are continuing to use this general roadmap:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>>> created a separate issue to track post-migration cleanup items:
>>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>>
>>>
>>> [1] https://github.com/apache/beam/pull/5054
>>> [2] https://github.com/apache/beam/pull/5057
>>> [3] https://github.com/apache/beam/pull/5087
>>> [4] https://github.com/apache/beam/pull/5098
>>> [5] https://github.com/apache/beam/pull/5047
>>> [6] https://github.com/apache/beam/pull/5088
>>> [7] https://github.com/apache/beam/pull/5086
>>> [8] https://github.com/apache/beam/pull/5066
>>> [9] https://github.com/apache/beam/pull/5059
>>> [10] https://github.com/apache/beam/pull/5045
>>> [11] https://github.com/apache/beam/pull/5091
>>> [12] https://github.com/apache/beam/pull/5093
>>> [13] https://github.com/apache/beam/pull/5069
>>> [14] https://github.com/apache/beam/pull/5068
>>> [15] https://beam.apache.org/contribute/contribution-guide/
>>> [16] https://github.com/apache/beam-site/pull/412
>>> [17] https://github.com/apache/beam-site/pull/414
>>> [18] https://github.com/apache/beam/pull/5051
>>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>>
>>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>>
 I wanted to start a thread to summarize the current state of Gradle
 migration. We've made lots of good progress so far this week. Here's the
 status from what I can tell-- please add or correct anything I missed:

 * Release artifacts can be built and published for Snapshot and
 officlal releases [1]
 * Gradle-generated releases have been validated with the the Apache
 Beam archetype generation quickstart; still needs additional validation.
 * Generated release pom files have correct project 

Re: Gradle Status [April 11]

[Nathan Fisher]

Is there a document or similar outlining the decision behind Gradle over
Bazel?

For purely intellectual curiosity I’m curious what tradeoffs/benefits were
considered when evaluating the two or if it was more a matter of
community/contributor familiarity.

I found this thread which seems to imply issues around dependency
management;

https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E

On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:

> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> *  Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items:
>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and officlal
>>> releases [1]
>>> * Gradle-generated releases have been validated with the the Apache Beam
>>> archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>> learn the new system-- please add your own. This will eventually feed into
>>> official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle instead
>>> of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) 

Re: Gradle Status [April 11]

[Kenneth Knowles]

Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
Netty has been introduced since yesterday. Etienne mentioned he has worked
toward setting up periodic runs on all runners, so this should help get us
towards that. We'll probably prefer to build standalone fat jars for
selected runners and use those, which is pending unknown issues in the
shadow config leaving out required dependencies.

Kenn

On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:

> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> *  Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3]