Fwd: Jenkins instability and restarts

[Kenneth Knowles]

-- Forwarded message --
From: Daniel Pono Takamori 
Date: Wed, Nov 1, 2017 at 8:36 PM
Subject: Jenkins instability and restarts
To: bui...@apache.org


Hello builds@ enthusiasts.

As I'm sure you are aware, we're encountering a rather nasty bug
affecting a small subset of jobs but in turn our fix is affecting the
entire Jenkins stack.  I've created
https://issues.apache.org/jira/browse/INFRA-15424 to track what we
know and will be compiling a bug report for the Jenkins JIRA.
To recap, Jenkins seems to be arbitrarily stop scheduling certain jobs
(Mesos, Beam, and Hive are the most affected).  And the only fix we've
found is to restart Jenkins, which is obviously less than ideal.
I apologize for the inconvenience this is causing and we hope to
resolve it, or at least make head way with it soon.

-Pono from Infra

Re: The trouble with DisplayData tests

[Pablo Estrada]

>
> As of today, the Java and Python Dataflow runners actually transmit the
> entire original pipeline, which has slots for display data on composites.
> So the future in which this is actually used - at least for Dataflow -
> might be very near.
>

I am not 100% sure, but I believe this is not true.
The workflow graph does not contain composites. It only contains
primitives, and dataflow uses their names to infer composites (e.g.
'Do/DoFirst' and 'Do/DoSecond' are inferred to belong to the same composite
because they share their first level translform).
Can someone confirm whether this is the case?
If so, the workflow graph representation for Dataflow would need to change
to explicitly include display data for composites.

Nonetheless, display data is very useful, so I'd think it's useful to make
it available in other runners, and for composites..

Best
-P.

>
>
> - There are a lot of tests testing display data of composite transforms -
> > these tests are currently testing something that is a no-op in
> production;
> > and these tests often are a maintenance burden when refactoring a
> > transform.
>
>
> Which composites' tests in particular? Are they legitimate tests?
>
>
> What should we do about this? I see two options:
> > - Publish guidance for library authors that Beam is not currently ready
> for
> > supporting HasDisplayData on composite transforms, and you shouldn't
> bother
> > implementing or testing it (though you can implement it on primitive
> > transforms of course - which is basically just Read.from(Source) and
> ParDo)
> > - Say that people should be implementing and testing this feature
> > nevertheless, and promise that Beam runners are going to make it "worth
> it"
> > (not be a no-op) in the foreseeable future.
> >
>
> Option 3: Don't say anyone _should_ do any particular thing, but promise
> that [some] Beam runners are [probably] going to make it "worth it" pretty
> soon. I expect third-party transform authors are already not using this,
> are they?
>
> Kenn
>
>
>
> > [1]
> > https://lists.apache.org/thread.html/53b33f17d981054ed198af03511969
> > 07bf147b6acf16160dbf395b62@%3Cdev.beam.apache.org%3E
> > [2] https://issues.apache.org/jira/browse/BEAM-366
> >
>

Re: Hello

[Ankur Goenka]

Thanks for adding me to the group.

On Wed, Nov 1, 2017 at 12:04 PM, Lukasz Cwik 
wrote:

> Welcome, you have been added.
>
> On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka 
> wrote:
>
> > Hi Guys,
> >
> > I recently joined Google and will be contributing to Apache Beam project.
> > Please add me to the apache jira. My jira id is angoenka
> >
> > Looking forward to work with you guys!
> >
> > Thanks,
> > Ankur
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Kenneth Knowles]

I have started one, here: https://github.com/kennknowles/beam/commits/bazel.
It is not nearly as far along as Luke's. For the POC I am just putting
things in one root BUILD, and learning where we might find the necessary
plugins as I go. I am happy to grant push access to this branch. It would
be superb if you had some time to work through the Python steps.

On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay 
wrote:

> Has anyone started a POC with Bazel? I would be interested in helping that
> effort.
>
> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik 
> wrote:
>
> > I have started a POC for using Gradle here:
> > https://github.com/lukecwik/incubator-beam/tree/gradle
> >
> > Things that work:
> > * compiling all Java code (src/main and src/test)
> > * generating source from protos
> > * generating source from avro
> > * running rat, checkstyle
> >
> > Partially working:
> > * generating maven pom (albeit with wrong dependencies for some
> > subprojects)
> > * running tests (~80% pass, remainder seem to be dependency related but
> are
> > uninvestigated)
> >
> > Things that don't work:
> > * anything Python/Go/Docker compilation related
> > * many tests fail because I messed up dependencies
> > * anything shading related
> > * minor plugins like eclipse code formatter/...
> > * running @NeedsRunner/@ValidatesRunner/integration tests
> >
> > Feel free to reach out to me on Slack if you would like to try to tackle
> a
> > piece of the POC to prevent duplication of effort from anyone working on
> > it.
> >
> >
> >
> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Agree to move forward on a PoC.
> > >
> > > Thanks Reuven for bringing discussion on the mailing list !
> > >
> > > Regards
> > > JB
> > >
> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> > > wrote:
> > > >Some good discussion here, and thanks to JB and Romain for adding to
> > > >it!
> > > >
> > > >JB makes the good point that we still need to release Maven artifacts,
> > > >as
> > > >many Beam users want to develop using Maven. So none of this
> discussion
> > > >will affect our release process, as we still need Maven "releases."
> > > >
> > > >At this point, if people are interested, I see no harm in prototyping.
> > > >Having working alternatives will give us a better basis for comparison
> > > >to
> > > >understand whether these other build systems give us anything over
> what
> > > >Maven does.
> > > >
> > > >Reuven
> > > >
> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen  >
> > > >wrote:
> > > >
> > > >> As a contributor to the Beam Python SDK, I noticed that many of the
> > > >points
> > > >> above regarding Maven and Gradle pertain mostly to Java SDK
> > > >development.
> > > >> For Python development, Maven is much less natural, and we end up
> > > >just
> > > >> shelling out to perform builds and tests.  For Python SDK (and
> > > >upcoming Go
> > > >> SDK development), an option to use Bazel would be quite useful.
> > > >>
> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> > > >>  wrote:
> > > >>
> > > >> > +1, Maven is both a build tool and a repository, and the latter is
> > > >> > essential to keep. Both Gradel and Bazel can interface with this
> > > >> > repository.
> > > >> >
> > > >> > I am, however, very supportive of moving away from Maven to a tool
> > > >> > that supports correct incremental, hermetic, dependency-driven,
> > > >> > multi-langauge, and hopefully fast builds for our own development.
> > > >> >
> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> > > >> >  wrote:
> > > >> > > Echoing what JB and Reuven said, we absolutely must provide
> maven
> > > >> central
> > > >> > > artifacts for Java users, just as we provide pypi artifacts for
> > > >Python
> > > >> > > users.
> > > >> > >
> > > >> > > I see Maven as still a viable tool for single-module Java
> builds,
> > > >> > > especially considering its rich plugin ecosystem.
> > > >> > >
> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > > > > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> I think that's a very good point. No matter what build system
> we
> > > >use
> > > >> for
> > > >> > >> our own personal development, we still need to release Maven
> > > >artifacts
> > > >> > and
> > > >> > >> releases as we need to support our users using Maven.
> > > >> > >>
> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> > > >> j...@nanthrax.net
> > > >> > >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > Generally speaking, it's interesting to evaluate
> alternatives,
> > > >> > especially
> > > >> > >> > Gradle. My point is also to keep Maven artifacts and
> > > >"releases" as
> > > >> > most
> > > >> > >> of
> > > >> > >> > our users will use Maven.
> > > >> > >> > For 

Re: The trouble with DisplayData tests

[Lukasz Cwik]

I think we should keep the tests around and encourage people to use it.

With the work that Ken put in to get the Pipeline proto to passed to the
Dataflow workers means that these composites will be flowing through
Dataflow to the execution side. I could see that this will eventually show
up in the Dataflow UI as well.

On Mon, Oct 30, 2017 at 1:45 PM, Eugene Kirpichov <
kirpic...@google.com.invalid> wrote:

> On Fri, Oct 27, 2017 at 12:40 AM Kenneth Knowles 
> wrote:
>
> > I see where you are coming from. It is truly a marginal feature of Beam
> at
> > the moment, but really *really* useful in debugging, when a runner takes
> > advantage of it. More inline - FWIW it may seem like I'm defending the
> > tests, but I have been in the same situation and felt the same way. I'm
> > just trying to ground things in specifics.
> >
> > On Thu, Oct 26, 2017 at 2:24 PM, Eugene Kirpichov <
> > kirpic...@google.com.invalid> wrote:
> >
> > > Hello,
> > >
> > > DisplayData is currently in an odd position:
> > > - Dataflow is the only runner that supports it at all [1]
> > > - Dataflow supports it only for primitive transforms, but discards it
> for
> > > composite transforms [2](which is quite problematic for very complex
> but
> > > commonly used composite transforms, such as TextIO.write() and
> > > BigQueryIO.write())
> > >
> >
> > As of today, the Java and Python Dataflow runners actually transmit the
> > entire original pipeline, which has slots for display data on composites.
> > So the future in which this is actually used - at least for Dataflow -
> > might be very near.
> >
> >
> > - There are a lot of tests testing display data of composite transforms -
> > > these tests are currently testing something that is a no-op in
> > production;
> > > and these tests often are a maintenance burden when refactoring a
> > > transform.
> >
> >
> > Which composites' tests in particular? Are they legitimate tests?
> >
> Pretty much all of them.
> For example:
> https://github.com/apache/beam/blob/master/sdks/java/
> core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java#L585
>
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-
>  google-cloud-platform/src/test/java/org/apache/beam/sdk/
> io/gcp/pubsub/PubsubIOTest.java#L98>
> platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/
> PubsubIOTest.java#L98
>  google-cloud-platform/src/test/java/org/apache/beam/sdk/
> io/gcp/pubsub/PubsubIOTest.java#L98>
>  and
> https://github.com/apache/beam/blob/master/sdks/java/io/
> google-cloud-platform/src/test/java/org/apache/beam/sdk/
> io/gcp/pubsub/PubsubIOTest.java#L222
>
>
> https://github.com/apache/beam/blob/master/sdks/java/
> core/src/test/java/org/apache/beam/sdk/transforms/
> FlatMapElementsTest.java#L169
>
>
> and so on. I think nearly all transforms in the SDK are composite, and have
> unit tests that test DisplayData of the transform itself, which is
> currently moot. Some have tests of "display data of primitive transforms"
> which is currently more useful in practice.
>
>
> >
> > What should we do about this? I see two options:
> > > - Publish guidance for library authors that Beam is not currently ready
> > for
> > > supporting HasDisplayData on composite transforms, and you shouldn't
> > bother
> > > implementing or testing it (though you can implement it on primitive
> > > transforms of course - which is basically just Read.from(Source) and
> > ParDo)
> > > - Say that people should be implementing and testing this feature
> > > nevertheless, and promise that Beam runners are going to make it "worth
> > it"
> > > (not be a no-op) in the foreseeable future.
> > >
> >
> > Option 3: Don't say anyone _should_ do any particular thing, but promise
> > that [some] Beam runners are [probably] going to make it "worth it"
> pretty
> > soon. I expect third-party transform authors are already not using this,
> > are they?
> >
> Most aren't, but some are [looking at sdks/java/io]:
> https://github.com/apache/beam/blob/master/sdks/java/io/
> hbase/src/test/java/org/apache/beam/sdk/io/hbase/HBaseIOTest.java#L328
>
> https://github.com/apache/beam/blob/master/sdks/java/io/
> tika/src/test/java/org/apache/beam/sdk/io/tika/TikaIOTest.java#L128
>
> https://github.com/apache/beam/blob/master/sdks/java/io/
> google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/
> DatastoreV1Test.java#L223
>
> I guess if Dataflow's (or another runner's) proper support of DisplayData
> on composite transforms is sufficiently near, or at least planned, then we
> should keep existing tests, and add more support and more tests once the
> support lands; but meanwhile not require or encourage new transform authors
> to implement it (but it's fine if they do). How does this sound?
>
>
> >
> > Kenn
> >
> >
> >
> > > [1]
> > > 

Re: Hello

[Lukasz Cwik]

Welcome, you have been added.

On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka 
wrote:

> Hi Guys,
>
> I recently joined Google and will be contributing to Apache Beam project.
> Please add me to the apache jira. My jira id is angoenka
>
> Looking forward to work with you guys!
>
> Thanks,
> Ankur
>

Re: Portability overview webpage

[Jean-Baptiste Onofré]

Thanks for the update. I will take a look.

Regards
JB

On Nov 1, 2017, 18:21, at 18:21, Henning Rohde  
wrote:
>Hi everyone,
>
>Although portability is a large and involved effort, it seems it
>doesn't
>have a high-level overview and plan written down anywhere. I added a
>proposed page with a 10,000 ft view and links to the webside under
>'Contribute (technical references)'. There is a page for ongoing
>projects,
>but portability is much more encompassing and seems to be more suited
>for
>it's own page.
>
>The PR is:
>
> https://github.com/apache/beam-site/pull/340
>
>I'm sending it out to the dev list for more visibility. Please let me
>know
>if you have any comments or objections, or if there is a better place
>for
>this content.
>
>Thanks,
> Henning

Portability overview webpage

[Henning Rohde]

Hi everyone,

Although portability is a large and involved effort, it seems it doesn't
have a high-level overview and plan written down anywhere. I added a
proposed page with a 10,000 ft view and links to the webside under
'Contribute (technical references)'. There is a page for ongoing projects,
but portability is much more encompassing and seems to be more suited for
it's own page.

The PR is:

 https://github.com/apache/beam-site/pull/340

I'm sending it out to the dev list for more visibility. Please let me know
if you have any comments or objections, or if there is a better place for
this content.

Thanks,
 Henning

Re: [DISCUSS] Move away from Apache Maven as build tool

[Ahmet Altay]

Has anyone started a POC with Bazel? I would be interested in helping that
effort.

On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik 
wrote:

> I have started a POC for using Gradle here:
> https://github.com/lukecwik/incubator-beam/tree/gradle
>
> Things that work:
> * compiling all Java code (src/main and src/test)
> * generating source from protos
> * generating source from avro
> * running rat, checkstyle
>
> Partially working:
> * generating maven pom (albeit with wrong dependencies for some
> subprojects)
> * running tests (~80% pass, remainder seem to be dependency related but are
> uninvestigated)
>
> Things that don't work:
> * anything Python/Go/Docker compilation related
> * many tests fail because I messed up dependencies
> * anything shading related
> * minor plugins like eclipse code formatter/...
> * running @NeedsRunner/@ValidatesRunner/integration tests
>
> Feel free to reach out to me on Slack if you would like to try to tackle a
> piece of the POC to prevent duplication of effort from anyone working on
> it.
>
>
>
> On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
> wrote:
>
> > Agree to move forward on a PoC.
> >
> > Thanks Reuven for bringing discussion on the mailing list !
> >
> > Regards
> > JB
> >
> > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> > wrote:
> > >Some good discussion here, and thanks to JB and Romain for adding to
> > >it!
> > >
> > >JB makes the good point that we still need to release Maven artifacts,
> > >as
> > >many Beam users want to develop using Maven. So none of this discussion
> > >will affect our release process, as we still need Maven "releases."
> > >
> > >At this point, if people are interested, I see no harm in prototyping.
> > >Having working alternatives will give us a better basis for comparison
> > >to
> > >understand whether these other build systems give us anything over what
> > >Maven does.
> > >
> > >Reuven
> > >
> > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
> > >wrote:
> > >
> > >> As a contributor to the Beam Python SDK, I noticed that many of the
> > >points
> > >> above regarding Maven and Gradle pertain mostly to Java SDK
> > >development.
> > >> For Python development, Maven is much less natural, and we end up
> > >just
> > >> shelling out to perform builds and tests.  For Python SDK (and
> > >upcoming Go
> > >> SDK development), an option to use Bazel would be quite useful.
> > >>
> > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> > >>  wrote:
> > >>
> > >> > +1, Maven is both a build tool and a repository, and the latter is
> > >> > essential to keep. Both Gradel and Bazel can interface with this
> > >> > repository.
> > >> >
> > >> > I am, however, very supportive of moving away from Maven to a tool
> > >> > that supports correct incremental, hermetic, dependency-driven,
> > >> > multi-langauge, and hopefully fast builds for our own development.
> > >> >
> > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> > >> >  wrote:
> > >> > > Echoing what JB and Reuven said, we absolutely must provide maven
> > >> central
> > >> > > artifacts for Java users, just as we provide pypi artifacts for
> > >Python
> > >> > > users.
> > >> > >
> > >> > > I see Maven as still a viable tool for single-module Java builds,
> > >> > > especially considering its rich plugin ecosystem.
> > >> > >
> > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > > > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> I think that's a very good point. No matter what build system we
> > >use
> > >> for
> > >> > >> our own personal development, we still need to release Maven
> > >artifacts
> > >> > and
> > >> > >> releases as we need to support our users using Maven.
> > >> > >>
> > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> > >> j...@nanthrax.net
> > >> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Generally speaking, it's interesting to evaluate alternatives,
> > >> > especially
> > >> > >> > Gradle. My point is also to keep Maven artifacts and
> > >"releases" as
> > >> > most
> > >> > >> of
> > >> > >> > our users will use Maven.
> > >> > >> > For incremental build, afair, there's some enhancements on
> > >Maven
> > >> but I
> > >> > >> > have to take a look.
> > >> > >> >
> > >> > >> > Regards
> > >> > >> > JB
> > >> > >> >
> > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> > >> > >> >  wrote:
> > >> > >> > >Hi!
> > >> > >> > >
> > >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't
> > >really
> > >> do
> > >> > >> > >incremental builds [1]. The best it can do is, it seems,
> > >recompile
> > >> > only
> > >> > >> > >changed files, but Java compilation is a tiny part of the
> > >overall
> > >> > >> > >build.
> > >> > >> > >
> > >> > >> > >Almost all time is taken by other plugins, 

Re: [VOTE] Switch to new JIRA workflow for pending proposals

[Chamikara Jayalath]

+1

Thanks,
Cham

On Wed, Nov 1, 2017 at 9:42 AM Ted Yu  wrote:

> +1
>  Original message From: Henning Rohde
>  Date: 11/1/17  9:29 AM  (GMT-08:00) To:
> dev@beam.apache.org Subject: Re: [VOTE] Switch to new JIRA workflow for
> pending proposals
> + 1. Also +1 to Kenn's suggestion of adding a "triage" stage prior to
> "open" for other bugs and features.
>
> On Wed, Nov 1, 2017 at 9:19 AM, Lukasz Cwik 
> wrote:
>
> > +1
> >
> > On Wed, Nov 1, 2017 at 5:52 AM, Kenneth Knowles 
> > wrote:
> >
> > > +1 I like it.
> > >
> > > This is the JIRA alternative to FLIP/KIP/HIP etc, yes? I definitely
> favor
> > > having automation around that rather than just wiki or web page
> entries.
> > >
> > > Any thoughts on how it interacts with other types of tasks?
> > >
> > > Also, an analogous initial state seems useful for all bugs & features.
> > > Namely, a "needs triage" state prior to "open".
> > >
> > > On Tue, Oct 31, 2017 at 10:23 PM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > It sounds like a good workflow to track the state of ideas/proposals.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Nov 1, 2017, 05:43, at 05:43, Reuven Lax  >
> > > > wrote:
> > > > >There is a new JIRA workflow for tracking pending proposals. This
> > > > >workflow
> > > > >is described in https://issues.apache.org/jira/browse/INFRA-12698.
> > > > >Please
> > > > >vote on whether you believe Beam should adopt this new workflow.
> > > > >
> > > > >Thanks,
> > > > >
> > > > >Reuven
> > > >
> > >
> >
>

Re: [VOTE] Switch to new JIRA workflow for pending proposals

[Henning Rohde]

+ 1. Also +1 to Kenn's suggestion of adding a "triage" stage prior to
"open" for other bugs and features.

On Wed, Nov 1, 2017 at 9:19 AM, Lukasz Cwik 
wrote:

> +1
>
> On Wed, Nov 1, 2017 at 5:52 AM, Kenneth Knowles 
> wrote:
>
> > +1 I like it.
> >
> > This is the JIRA alternative to FLIP/KIP/HIP etc, yes? I definitely favor
> > having automation around that rather than just wiki or web page entries.
> >
> > Any thoughts on how it interacts with other types of tasks?
> >
> > Also, an analogous initial state seems useful for all bugs & features.
> > Namely, a "needs triage" state prior to "open".
> >
> > On Tue, Oct 31, 2017 at 10:23 PM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > +1
> > >
> > > It sounds like a good workflow to track the state of ideas/proposals.
> > >
> > > Regards
> > > JB
> > >
> > > On Nov 1, 2017, 05:43, at 05:43, Reuven Lax 
> > > wrote:
> > > >There is a new JIRA workflow for tracking pending proposals. This
> > > >workflow
> > > >is described in https://issues.apache.org/jira/browse/INFRA-12698.
> > > >Please
> > > >vote on whether you believe Beam should adopt this new workflow.
> > > >
> > > >Thanks,
> > > >
> > > >Reuven
> > >
> >
>

Re: [DISCUSS] Move away from Apache Maven as build tool

[Lukasz Cwik]

I have started a POC for using Gradle here:
https://github.com/lukecwik/incubator-beam/tree/gradle

Things that work:
* compiling all Java code (src/main and src/test)
* generating source from protos
* generating source from avro
* running rat, checkstyle

Partially working:
* generating maven pom (albeit with wrong dependencies for some subprojects)
* running tests (~80% pass, remainder seem to be dependency related but are
uninvestigated)

Things that don't work:
* anything Python/Go/Docker compilation related
* many tests fail because I messed up dependencies
* anything shading related
* minor plugins like eclipse code formatter/...
* running @NeedsRunner/@ValidatesRunner/integration tests

Feel free to reach out to me on Slack if you would like to try to tackle a
piece of the POC to prevent duplication of effort from anyone working on it.



On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré 
wrote:

> Agree to move forward on a PoC.
>
> Thanks Reuven for bringing discussion on the mailing list !
>
> Regards
> JB
>
> On Nov 1, 2017, 03:20, at 03:20, Reuven Lax 
> wrote:
> >Some good discussion here, and thanks to JB and Romain for adding to
> >it!
> >
> >JB makes the good point that we still need to release Maven artifacts,
> >as
> >many Beam users want to develop using Maven. So none of this discussion
> >will affect our release process, as we still need Maven "releases."
> >
> >At this point, if people are interested, I see no harm in prototyping.
> >Having working alternatives will give us a better basis for comparison
> >to
> >understand whether these other build systems give us anything over what
> >Maven does.
> >
> >Reuven
> >
> >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen 
> >wrote:
> >
> >> As a contributor to the Beam Python SDK, I noticed that many of the
> >points
> >> above regarding Maven and Gradle pertain mostly to Java SDK
> >development.
> >> For Python development, Maven is much less natural, and we end up
> >just
> >> shelling out to perform builds and tests.  For Python SDK (and
> >upcoming Go
> >> SDK development), an option to use Bazel would be quite useful.
> >>
> >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw
> >>  wrote:
> >>
> >> > +1, Maven is both a build tool and a repository, and the latter is
> >> > essential to keep. Both Gradel and Bazel can interface with this
> >> > repository.
> >> >
> >> > I am, however, very supportive of moving away from Maven to a tool
> >> > that supports correct incremental, hermetic, dependency-driven,
> >> > multi-langauge, and hopefully fast builds for our own development.
> >> >
> >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles
> >> >  wrote:
> >> > > Echoing what JB and Reuven said, we absolutely must provide maven
> >> central
> >> > > artifacts for Java users, just as we provide pypi artifacts for
> >Python
> >> > > users.
> >> > >
> >> > > I see Maven as still a viable tool for single-module Java builds,
> >> > > especially considering its rich plugin ecosystem.
> >> > >
> >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax
> > >> >
> >> > > wrote:
> >> > >
> >> > >> I think that's a very good point. No matter what build system we
> >use
> >> for
> >> > >> our own personal development, we still need to release Maven
> >artifacts
> >> > and
> >> > >> releases as we need to support our users using Maven.
> >> > >>
> >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste Onofré <
> >> j...@nanthrax.net
> >> > >
> >> > >> wrote:
> >> > >>
> >> > >> > Generally speaking, it's interesting to evaluate alternatives,
> >> > especially
> >> > >> > Gradle. My point is also to keep Maven artifacts and
> >"releases" as
> >> > most
> >> > >> of
> >> > >> > our users will use Maven.
> >> > >> > For incremental build, afair, there's some enhancements on
> >Maven
> >> but I
> >> > >> > have to take a look.
> >> > >> >
> >> > >> > Regards
> >> > >> > JB
> >> > >> >
> >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov
> >> > >> >  wrote:
> >> > >> > >Hi!
> >> > >> > >
> >> > >> > >Many of these points sound valid, but AFAICT Maven doesn't
> >really
> >> do
> >> > >> > >incremental builds [1]. The best it can do is, it seems,
> >recompile
> >> > only
> >> > >> > >changed files, but Java compilation is a tiny part of the
> >overall
> >> > >> > >build.
> >> > >> > >
> >> > >> > >Almost all time is taken by other plugins, such as unit
> >testing or
> >> > >> > >findbugs
> >> > >> > >- and Maven does not seem to currently support features such
> >as "do
> >> > not
> >> > >> > >rerun unit tests of a module if the code didn't change".
> >> > >> > >
> >> > >> > >The fact that the surefire plugin has existed for >11 years
> >> (version
> >> > >> > >2.0
> >> > >> > >was released in 2006) and still doesn't have this feature
> >makes me
> >> > >> > >think
> 

Re: [VOTE] Switch to new JIRA workflow for pending proposals

[Kenneth Knowles]

+1 I like it.

This is the JIRA alternative to FLIP/KIP/HIP etc, yes? I definitely favor
having automation around that rather than just wiki or web page entries.

Any thoughts on how it interacts with other types of tasks?

Also, an analogous initial state seems useful for all bugs & features.
Namely, a "needs triage" state prior to "open".

On Tue, Oct 31, 2017 at 10:23 PM, Jean-Baptiste Onofré 
wrote:

> +1
>
> It sounds like a good workflow to track the state of ideas/proposals.
>
> Regards
> JB
>
> On Nov 1, 2017, 05:43, at 05:43, Reuven Lax 
> wrote:
> >There is a new JIRA workflow for tracking pending proposals. This
> >workflow
> >is described in https://issues.apache.org/jira/browse/INFRA-12698.
> >Please
> >vote on whether you believe Beam should adopt this new workflow.
> >
> >Thanks,
> >
> >Reuven
>