Re: [discuss] java profile

[Romain Manni-Bucau]

Le 29 nov. 2017 18:48, "Lukasz Cwik"  a écrit :

In practice you can get incremental builds, its not just theory.


:) give it a try matching all dev build habits.

Current gradle setup already breaks it setting up the whole tree for
subprojects, same will apply for incremental setup and is boring as well as
ut slows down the dev workflow for cli driven dev.



On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau 
wrote:

>
>
> Le 29 nov. 2017 01:29, "Robert Bradshaw"  a écrit :
>
> I am curious what you mean by the "Python Build" as by nature there's
> no (significant) compilation that happens. Python tests, on the other
> hands, are completely sequential (and very redundant--each test file
> is run at least three times). With the potential of Java builds
> becoming fast enough that Python becomes the long (or at least a
> significant) factor, I'm staring to get motivated to look at this more
> closely :).
>
>
> The python generated and executed code. I didnt look more than that bit it
> looks like setup (pip) and tests are not that fast even if several are
> skipped (?? A missing dep shouldnt reduce the coverage right?)
>
>
> With respect to correct incremental builds, not only is it possible,
> but it can be done at scale (e.g.
> http://google-engtools.blogspot.com/2011/08/build-in-cloud-h
> ow-build-system-works.html
> https://blog.bazel.build/2017/08/25/introducing-sandboxfs.html
> https://nixos.org/nix/) One trick is to disallow tools to even look at
> inputs that are not declared. As you've probably guessed, I'm partial
> to reproducible builds (and reproducible research in general) myself.
> Unfortunately the explorations into bazel seem to indicate that the
> third-party dependency system isn't sufficiently mature (vs. the
> one-codebase environment it was developed in).
>
>
> It is a theory yes but in practise for a multimodule project you have to
> not respect it. It only impacts you with indirect code path like SPI or
> reflection. Beam does the first one at least.
>
> Dont get me wrong, it is not a blocker but it is also not magic and you
> will need to add a no daemon and clean build to avoid caching issues it is
> easy to have with gradle in dev mode.
>
>
> With regards to the original topic, hopefully the build system can
> just take care of testing exactly what needs to be tested based on
> changes. Note that this will likely cross language barriers more and
> more as the portability work progresses (e.g. we'll have
> Python-on-Flink tests that should be run if either the Python SDK or
> the Flink runner gets modified).
>
>
>
> The git integration of maven we talked earlier does that. I guess/hope
> gradle has the same but it is bound to the scm rather than the build tool.
> Worse case a git (status based) script should enable to launch that.
>
>
> - Robert
>
>
>
> On Tue, Nov 28, 2017 at 12:49 PM, Romain Manni-Bucau
>  wrote:
> > Lukasz: only for an isolated "system" which is a module - assuming you
> still
> > want to be able to build a submodule without building and revalidating
> the
> > whole tree which is important in dev IMO. This means you shouldnt handle
> > inputs outside "current" module and therefore miss easily some (typically
> > test) execution/coverage - incremental compilation/style check etc is not
> > that costly compared to tests. So this is far to be as straight forward
> as
> > it looks if you want to keep the same guarantees. Once again for
> > checkstyle/findbugs/javac/rat etc this is trivial - and even with maven
> > actually - but doesnt save a significant amount of time in dev whereas
> tests
> > do.
> >
> >
> > Le 28 nov. 2017 20:11, "Kenneth Knowles"  a écrit :
> >
> > I seem to remember a tool called `make` that was pretty good at this.
> >
> > On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik  wrote:
> >>
> >> Its been well shown that a build system that uses input/output set
> change
> >> detection can correctly implement incremental builds. Build systems are
> not
> >> tied to knowing the internal details of how Java compiles things.
> Knowing
> >> that there are some inputs, a process, and some outputs is enough to
> know
> >> when the process needs to be rerun.
> >>
> >> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau
> >>  wrote:
> >>>
> >>> Hmm, no.
> >>>
> >>> Incremental build is never correctly implemented cause there is just no
> >>> way to detect some dependencies statically with java code - or any
> dynamic
> >>> language.
> >>>
> >>> Side note: same applies for gradle daemon usage BTW.
> >>>
> >>> After if the list is not maintained it is a bug at the same level than
> >>> coding a toString() with "null.toString()". This is not very hard to
> handle
> >>> the list of modules and worse case a mvnextension can make it coded if
> you
> >>> feel more comfortable with this kind of solution.
> >>>
> >>> Le 27 

Re: [discuss] java profile

[Lukasz Cwik]

In practice you can get incremental builds, its not just theory.

On Tue, Nov 28, 2017 at 10:12 PM, Romain Manni-Bucau 
wrote:

>
>
> Le 29 nov. 2017 01:29, "Robert Bradshaw"  a écrit :
>
> I am curious what you mean by the "Python Build" as by nature there's
> no (significant) compilation that happens. Python tests, on the other
> hands, are completely sequential (and very redundant--each test file
> is run at least three times). With the potential of Java builds
> becoming fast enough that Python becomes the long (or at least a
> significant) factor, I'm staring to get motivated to look at this more
> closely :).
>
>
> The python generated and executed code. I didnt look more than that bit it
> looks like setup (pip) and tests are not that fast even if several are
> skipped (?? A missing dep shouldnt reduce the coverage right?)
>
>
> With respect to correct incremental builds, not only is it possible,
> but it can be done at scale (e.g.
> http://google-engtools.blogspot.com/2011/08/build-in-cloud-
> how-build-system-works.html
> https://blog.bazel.build/2017/08/25/introducing-sandboxfs.html
> https://nixos.org/nix/) One trick is to disallow tools to even look at
> inputs that are not declared. As you've probably guessed, I'm partial
> to reproducible builds (and reproducible research in general) myself.
> Unfortunately the explorations into bazel seem to indicate that the
> third-party dependency system isn't sufficiently mature (vs. the
> one-codebase environment it was developed in).
>
>
> It is a theory yes but in practise for a multimodule project you have to
> not respect it. It only impacts you with indirect code path like SPI or
> reflection. Beam does the first one at least.
>
> Dont get me wrong, it is not a blocker but it is also not magic and you
> will need to add a no daemon and clean build to avoid caching issues it is
> easy to have with gradle in dev mode.
>
>
> With regards to the original topic, hopefully the build system can
> just take care of testing exactly what needs to be tested based on
> changes. Note that this will likely cross language barriers more and
> more as the portability work progresses (e.g. we'll have
> Python-on-Flink tests that should be run if either the Python SDK or
> the Flink runner gets modified).
>
>
>
> The git integration of maven we talked earlier does that. I guess/hope
> gradle has the same but it is bound to the scm rather than the build tool.
> Worse case a git (status based) script should enable to launch that.
>
>
> - Robert
>
>
>
> On Tue, Nov 28, 2017 at 12:49 PM, Romain Manni-Bucau
>  wrote:
> > Lukasz: only for an isolated "system" which is a module - assuming you
> still
> > want to be able to build a submodule without building and revalidating
> the
> > whole tree which is important in dev IMO. This means you shouldnt handle
> > inputs outside "current" module and therefore miss easily some (typically
> > test) execution/coverage - incremental compilation/style check etc is not
> > that costly compared to tests. So this is far to be as straight forward
> as
> > it looks if you want to keep the same guarantees. Once again for
> > checkstyle/findbugs/javac/rat etc this is trivial - and even with maven
> > actually - but doesnt save a significant amount of time in dev whereas
> tests
> > do.
> >
> >
> > Le 28 nov. 2017 20:11, "Kenneth Knowles"  a écrit :
> >
> > I seem to remember a tool called `make` that was pretty good at this.
> >
> > On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik  wrote:
> >>
> >> Its been well shown that a build system that uses input/output set
> change
> >> detection can correctly implement incremental builds. Build systems are
> not
> >> tied to knowing the internal details of how Java compiles things.
> Knowing
> >> that there are some inputs, a process, and some outputs is enough to
> know
> >> when the process needs to be rerun.
> >>
> >> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau
> >>  wrote:
> >>>
> >>> Hmm, no.
> >>>
> >>> Incremental build is never correctly implemented cause there is just no
> >>> way to detect some dependencies statically with java code - or any
> dynamic
> >>> language.
> >>>
> >>> Side note: same applies for gradle daemon usage BTW.
> >>>
> >>> After if the list is not maintained it is a bug at the same level than
> >>> coding a toString() with "null.toString()". This is not very hard to
> handle
> >>> the list of modules and worse case a mvnextension can make it coded if
> you
> >>> feel more comfortable with this kind of solution.
> >>>
> >>> Le 27 nov. 2017 23:12, "Lukasz Cwik"  a
> écrit :
> 
>  Manually whitelisting/blacklisting sub-modules is error prone since it
>  hides issues due to incorrectly maintaining that list is the same
>  argument
>  as if the build process doesn't correctly invoke an incremental 

Re: [discuss] java profile

[Romain Manni-Bucau]

Lukasz: only for an isolated "system" which is a module - assuming you
still want to be able to build a submodule without building and
revalidating the whole tree which is important in dev IMO. This means you
shouldnt handle inputs outside "current" module and therefore miss easily
some (typically test) execution/coverage - incremental compilation/style
check etc is not that costly compared to tests. So this is far to be as
straight forward as it looks if you want to keep the same guarantees. Once
again for checkstyle/findbugs/javac/rat etc this is trivial - and even with
maven actually - but doesnt save a significant amount of time in dev
whereas tests do.

Le 28 nov. 2017 20:11, "Kenneth Knowles"  a écrit :

I seem to remember a tool called `make` that was pretty good at this.

On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik  wrote:

> Its been well shown that a build system that uses input/output set change
> detection can correctly implement incremental builds. Build systems are not
> tied to knowing the internal details of how Java compiles things. Knowing
> that there are some inputs, a process, and some outputs is enough to know
> when the process needs to be rerun.
>
> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau  > wrote:
>
>> Hmm, no.
>>
>> Incremental build is never correctly implemented cause there is just no
>> way to detect some dependencies statically with java code - or any dynamic
>> language.
>>
>> Side note: same applies for gradle daemon usage BTW.
>>
>> After if the list is not maintained it is a bug at the same level than
>> coding a toString() with "null.toString()". This is not very hard to handle
>> the list of modules and worse case a mvnextension can make it coded if you
>> feel more comfortable with this kind of solution.
>>
>> Le 27 nov. 2017 23:12, "Lukasz Cwik"  a écrit :
>>
>>> Manually whitelisting/blacklisting sub-modules is error prone since it
>>> hides issues due to incorrectly maintaining that list is the same
>>> argument
>>> as if the build process doesn't correctly invoke an incremental build
>>> process.
>>>
>>> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com>
>>> wrote:
>>>
>>> > Well for validation builds- pre PR, incremental support is pointless
>>> since
>>> > it easily hides issues die to caching so a solution saving half of the
>>> > build without loosing anuyhing would still be good IMHO.
>>> >
>>> > Le 27 nov. 2017 21:12, "Lukasz Cwik"  a
>>> écrit :
>>> >
>>> > > Incremental builds aren't correctly setup right now so your likely
>>> to see
>>> > > Python/Go rebuild even if there were no changes. See
>>> > > https://issues.apache.org/jira/browse/BEAM-3253
>>> > >
>>> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau <
>>> > > rmannibu...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > that was the goal: validate there was no side effect of the
>>> changes on
>>> > > > the whole project. Now the "not java" part of the build will not be
>>> > > > impacted by java changed so this is the part i want to skip since
>>> it
>>> > > > takes a lot of time and I have guarantees it is safe to skip them.
>>> > > >
>>> > > > Romain Manni-Bucau
>>> > > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > >
>>> > > >
>>> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
>>> > > > > Romain, that will build the entire project. I think you want to
>>> > execute
>>> > > > > (from the root of the project):
>>> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build
>>> > > > >
>>> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
>>> > > > rmannibu...@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > >> gradle build --no-daemon
>>> > > > >>
>>> > > > >> (with gradle 4.2)
>>> > > > >>
>>> > > > >> Romain Manni-Bucau
>>> > > > >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > > >>
>>> > > > >>
>>> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles
>>> >> > >:
>>> > > > >> > What is the gradle command you are using to build just the
>>> Python
>>> > > SDK?
>>> > > > >> >
>>> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
>>> > > > >> rmannibu...@gmail.com>
>>> > > > >> > wrote:
>>> > > > >> >
>>> > > > >> >> Hmm,
>>> > > > >> >>
>>> > > > >> >> issue is the same with gradle (locally python build takes
>>> 15mn
>>> > > alone
>>> > > > >> >> which is as much as the java build and it is not
>>> parallelized I
>>> > > > think)
>>> > > > >> >>
>>> > > > >> >> pl is not as smooth since it means doing it on each command
>>> > whereas
>>> > > > >> >> the proposal is automatically activated through settings.xml
>>> > > > >> >>
>>> > > > >> >> Romain Manni-Bucau
>>> > > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > > >> >>
>>> > > > >> >>
>>> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles
>>> > 

Re: [discuss] java profile

[Kenneth Knowles]

I seem to remember a tool called `make` that was pretty good at this.

On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik  wrote:

> Its been well shown that a build system that uses input/output set change
> detection can correctly implement incremental builds. Build systems are not
> tied to knowing the internal details of how Java compiles things. Knowing
> that there are some inputs, a process, and some outputs is enough to know
> when the process needs to be rerun.
>
> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau  > wrote:
>
>> Hmm, no.
>>
>> Incremental build is never correctly implemented cause there is just no
>> way to detect some dependencies statically with java code - or any dynamic
>> language.
>>
>> Side note: same applies for gradle daemon usage BTW.
>>
>> After if the list is not maintained it is a bug at the same level than
>> coding a toString() with "null.toString()". This is not very hard to handle
>> the list of modules and worse case a mvnextension can make it coded if you
>> feel more comfortable with this kind of solution.
>>
>> Le 27 nov. 2017 23:12, "Lukasz Cwik"  a écrit :
>>
>>> Manually whitelisting/blacklisting sub-modules is error prone since it
>>> hides issues due to incorrectly maintaining that list is the same
>>> argument
>>> as if the build process doesn't correctly invoke an incremental build
>>> process.
>>>
>>> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com>
>>> wrote:
>>>
>>> > Well for validation builds- pre PR, incremental support is pointless
>>> since
>>> > it easily hides issues die to caching so a solution saving half of the
>>> > build without loosing anuyhing would still be good IMHO.
>>> >
>>> > Le 27 nov. 2017 21:12, "Lukasz Cwik"  a
>>> écrit :
>>> >
>>> > > Incremental builds aren't correctly setup right now so your likely
>>> to see
>>> > > Python/Go rebuild even if there were no changes. See
>>> > > https://issues.apache.org/jira/browse/BEAM-3253
>>> > >
>>> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau <
>>> > > rmannibu...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > that was the goal: validate there was no side effect of the
>>> changes on
>>> > > > the whole project. Now the "not java" part of the build will not be
>>> > > > impacted by java changed so this is the part i want to skip since
>>> it
>>> > > > takes a lot of time and I have guarantees it is safe to skip them.
>>> > > >
>>> > > > Romain Manni-Bucau
>>> > > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > >
>>> > > >
>>> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
>>> > > > > Romain, that will build the entire project. I think you want to
>>> > execute
>>> > > > > (from the root of the project):
>>> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build
>>> > > > >
>>> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
>>> > > > rmannibu...@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > >> gradle build --no-daemon
>>> > > > >>
>>> > > > >> (with gradle 4.2)
>>> > > > >>
>>> > > > >> Romain Manni-Bucau
>>> > > > >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > > >>
>>> > > > >>
>>> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles
>>> >> > >:
>>> > > > >> > What is the gradle command you are using to build just the
>>> Python
>>> > > SDK?
>>> > > > >> >
>>> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
>>> > > > >> rmannibu...@gmail.com>
>>> > > > >> > wrote:
>>> > > > >> >
>>> > > > >> >> Hmm,
>>> > > > >> >>
>>> > > > >> >> issue is the same with gradle (locally python build takes
>>> 15mn
>>> > > alone
>>> > > > >> >> which is as much as the java build and it is not
>>> parallelized I
>>> > > > think)
>>> > > > >> >>
>>> > > > >> >> pl is not as smooth since it means doing it on each command
>>> > whereas
>>> > > > >> >> the proposal is automatically activated through settings.xml
>>> > > > >> >>
>>> > > > >> >> Romain Manni-Bucau
>>> > > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>> > > > >> >>
>>> > > > >> >>
>>> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles
>>> > >> > > >:
>>> > > > >> >> > I think you can already mostly do this with mvn -pl
>>> sdks/XYZ
>>> > -am
>>> > > > >> -amd. I
>>> > > > >> >> > think that we have other work (gradle support) underway
>>> that
>>> > will
>>> > > > make
>>> > > > >> >> this
>>> > > > >> >> > a non-issue since gradle automatically does even better
>>> than
>>> > the
>>> > > > >> profile
>>> > > > >> >> or
>>> > > > >> >> > -am -amd.
>>> > > > >> >> >
>>> > > > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
>>> > > > >> >> rmannibu...@gmail.com>
>>> > > > >> >> > wrote:
>>> > > > >> >> >
>>> > > > >> >> >> Hi guys,
>>> > > > >> >> >>
>>> > > > >> >> >> java/python/go/xxx support is great but as a developer you
>>> > > 

Re: [discuss] java profile

[Lukasz Cwik]

Its been well shown that a build system that uses input/output set change
detection can correctly implement incremental builds. Build systems are not
tied to knowing the internal details of how Java compiles things. Knowing
that there are some inputs, a process, and some outputs is enough to know
when the process needs to be rerun.

On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau 
wrote:

> Hmm, no.
>
> Incremental build is never correctly implemented cause there is just no
> way to detect some dependencies statically with java code - or any dynamic
> language.
>
> Side note: same applies for gradle daemon usage BTW.
>
> After if the list is not maintained it is a bug at the same level than
> coding a toString() with "null.toString()". This is not very hard to handle
> the list of modules and worse case a mvnextension can make it coded if you
> feel more comfortable with this kind of solution.
>
> Le 27 nov. 2017 23:12, "Lukasz Cwik"  a écrit :
>
>> Manually whitelisting/blacklisting sub-modules is error prone since it
>> hides issues due to incorrectly maintaining that list is the same argument
>> as if the build process doesn't correctly invoke an incremental build
>> process.
>>
>> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> wrote:
>>
>> > Well for validation builds- pre PR, incremental support is pointless
>> since
>> > it easily hides issues die to caching so a solution saving half of the
>> > build without loosing anuyhing would still be good IMHO.
>> >
>> > Le 27 nov. 2017 21:12, "Lukasz Cwik"  a
>> écrit :
>> >
>> > > Incremental builds aren't correctly setup right now so your likely to
>> see
>> > > Python/Go rebuild even if there were no changes. See
>> > > https://issues.apache.org/jira/browse/BEAM-3253
>> > >
>> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau <
>> > > rmannibu...@gmail.com>
>> > > wrote:
>> > >
>> > > > that was the goal: validate there was no side effect of the changes
>> on
>> > > > the whole project. Now the "not java" part of the build will not be
>> > > > impacted by java changed so this is the part i want to skip since it
>> > > > takes a lot of time and I have guarantees it is safe to skip them.
>> > > >
>> > > > Romain Manni-Bucau
>> > > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> > > >
>> > > >
>> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
>> > > > > Romain, that will build the entire project. I think you want to
>> > execute
>> > > > > (from the root of the project):
>> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build
>> > > > >
>> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
>> > > > rmannibu...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > >> gradle build --no-daemon
>> > > > >>
>> > > > >> (with gradle 4.2)
>> > > > >>
>> > > > >> Romain Manni-Bucau
>> > > > >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> > > > >>
>> > > > >>
>> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles
>> > > >:
>> > > > >> > What is the gradle command you are using to build just the
>> Python
>> > > SDK?
>> > > > >> >
>> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
>> > > > >> rmannibu...@gmail.com>
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> >> Hmm,
>> > > > >> >>
>> > > > >> >> issue is the same with gradle (locally python build takes 15mn
>> > > alone
>> > > > >> >> which is as much as the java build and it is not parallelized
>> I
>> > > > think)
>> > > > >> >>
>> > > > >> >> pl is not as smooth since it means doing it on each command
>> > whereas
>> > > > >> >> the proposal is automatically activated through settings.xml
>> > > > >> >>
>> > > > >> >> Romain Manni-Bucau
>> > > > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> > > > >> >>
>> > > > >> >>
>> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles
>> > > > > >:
>> > > > >> >> > I think you can already mostly do this with mvn -pl sdks/XYZ
>> > -am
>> > > > >> -amd. I
>> > > > >> >> > think that we have other work (gradle support) underway that
>> > will
>> > > > make
>> > > > >> >> this
>> > > > >> >> > a non-issue since gradle automatically does even better than
>> > the
>> > > > >> profile
>> > > > >> >> or
>> > > > >> >> > -am -amd.
>> > > > >> >> >
>> > > > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
>> > > > >> >> rmannibu...@gmail.com>
>> > > > >> >> > wrote:
>> > > > >> >> >
>> > > > >> >> >> Hi guys,
>> > > > >> >> >>
>> > > > >> >> >> java/python/go/xxx support is great but as a developer you
>> > > rarely
>> > > > >> hack
>> > > > >> >> >> on them all.
>> > > > >> >> >>
>> > > > >> >> >> For that reason I opened https://github.com/apache/
>> > > beam/pull/4173
>> > > > .
>> > > > >> >> >>
>> > > > >> >> >> Goal is to give each developer a way to build the whole
>> > project
>> > > > and
>> > > > >> 

Re: [discuss] java profile

[Romain Manni-Bucau]

Well for validation builds- pre PR, incremental support is pointless since
it easily hides issues die to caching so a solution saving half of the
build without loosing anuyhing would still be good IMHO.

Le 27 nov. 2017 21:12, "Lukasz Cwik"  a écrit :

> Incremental builds aren't correctly setup right now so your likely to see
> Python/Go rebuild even if there were no changes. See
> https://issues.apache.org/jira/browse/BEAM-3253
>
> On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> wrote:
>
> > that was the goal: validate there was no side effect of the changes on
> > the whole project. Now the "not java" part of the build will not be
> > impacted by java changed so this is the part i want to skip since it
> > takes a lot of time and I have guarantees it is safe to skip them.
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >
> >
> > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
> > > Romain, that will build the entire project. I think you want to execute
> > > (from the root of the project):
> > > ./gradlew :beam-sdks-parent:beam-sdks-python:build
> > >
> > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
> > rmannibu...@gmail.com>
> > > wrote:
> > >
> > >> gradle build --no-daemon
> > >>
> > >> (with gradle 4.2)
> > >>
> > >> Romain Manni-Bucau
> > >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >>
> > >>
> > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles :
> > >> > What is the gradle command you are using to build just the Python
> SDK?
> > >> >
> > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
> > >> rmannibu...@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hmm,
> > >> >>
> > >> >> issue is the same with gradle (locally python build takes 15mn
> alone
> > >> >> which is as much as the java build and it is not parallelized I
> > think)
> > >> >>
> > >> >> pl is not as smooth since it means doing it on each command whereas
> > >> >> the proposal is automatically activated through settings.xml
> > >> >>
> > >> >> Romain Manni-Bucau
> > >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >>
> > >> >>
> > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles  >:
> > >> >> > I think you can already mostly do this with mvn -pl sdks/XYZ -am
> > >> -amd. I
> > >> >> > think that we have other work (gradle support) underway that will
> > make
> > >> >> this
> > >> >> > a non-issue since gradle automatically does even better than the
> > >> profile
> > >> >> or
> > >> >> > -am -amd.
> > >> >> >
> > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
> > >> >> rmannibu...@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> >> Hi guys,
> > >> >> >>
> > >> >> >> java/python/go/xxx support is great but as a developer you
> rarely
> > >> hack
> > >> >> >> on them all.
> > >> >> >>
> > >> >> >> For that reason I opened https://github.com/apache/
> beam/pull/4173
> > .
> > >> >> >>
> > >> >> >> Goal is to give each developer a way to build the whole project
> > and
> > >> >> >> all the code he can impact at once but without caring of the
> code
> > he
> > >> >> >> doesn't modify at all - other languages.
> > >> >> >>
> > >> >> >> Wdyt?
> > >> >> >>
> > >> >> >> Romain Manni-Bucau
> > >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> > >> >> >>
> > >> >>
> > >>
> >
>

Re: [discuss] java profile

[Lukasz Cwik]

Incremental builds aren't correctly setup right now so your likely to see
Python/Go rebuild even if there were no changes. See
https://issues.apache.org/jira/browse/BEAM-3253

On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau 
wrote:

> that was the goal: validate there was no side effect of the changes on
> the whole project. Now the "not java" part of the build will not be
> impacted by java changed so this is the part i want to skip since it
> takes a lot of time and I have guarantees it is safe to skip them.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
> > Romain, that will build the entire project. I think you want to execute
> > (from the root of the project):
> > ./gradlew :beam-sdks-parent:beam-sdks-python:build
> >
> > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> gradle build --no-daemon
> >>
> >> (with gradle 4.2)
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles :
> >> > What is the gradle command you are using to build just the Python SDK?
> >> >
> >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
> >> rmannibu...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hmm,
> >> >>
> >> >> issue is the same with gradle (locally python build takes 15mn alone
> >> >> which is as much as the java build and it is not parallelized I
> think)
> >> >>
> >> >> pl is not as smooth since it means doing it on each command whereas
> >> >> the proposal is automatically activated through settings.xml
> >> >>
> >> >> Romain Manni-Bucau
> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >>
> >> >>
> >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
> >> >> > I think you can already mostly do this with mvn -pl sdks/XYZ -am
> >> -amd. I
> >> >> > think that we have other work (gradle support) underway that will
> make
> >> >> this
> >> >> > a non-issue since gradle automatically does even better than the
> >> profile
> >> >> or
> >> >> > -am -amd.
> >> >> >
> >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
> >> >> rmannibu...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi guys,
> >> >> >>
> >> >> >> java/python/go/xxx support is great but as a developer you rarely
> >> hack
> >> >> >> on them all.
> >> >> >>
> >> >> >> For that reason I opened https://github.com/apache/beam/pull/4173
> .
> >> >> >>
> >> >> >> Goal is to give each developer a way to build the whole project
> and
> >> >> >> all the code he can impact at once but without caring of the code
> he
> >> >> >> doesn't modify at all - other languages.
> >> >> >>
> >> >> >> Wdyt?
> >> >> >>
> >> >> >> Romain Manni-Bucau
> >> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >> >>
> >> >>
> >>
>

Re: [discuss] java profile

[Romain Manni-Bucau]

that was the goal: validate there was no side effect of the changes on
the whole project. Now the "not java" part of the build will not be
impacted by java changed so this is the part i want to skip since it
takes a lot of time and I have guarantees it is safe to skip them.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-27 20:28 GMT+01:00 Lukasz Cwik :
> Romain, that will build the entire project. I think you want to execute
> (from the root of the project):
> ./gradlew :beam-sdks-parent:beam-sdks-python:build
>
> On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau 
> wrote:
>
>> gradle build --no-daemon
>>
>> (with gradle 4.2)
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles :
>> > What is the gradle command you are using to build just the Python SDK?
>> >
>> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> >> Hmm,
>> >>
>> >> issue is the same with gradle (locally python build takes 15mn alone
>> >> which is as much as the java build and it is not parallelized I think)
>> >>
>> >> pl is not as smooth since it means doing it on each command whereas
>> >> the proposal is automatically activated through settings.xml
>> >>
>> >> Romain Manni-Bucau
>> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >>
>> >>
>> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
>> >> > I think you can already mostly do this with mvn -pl sdks/XYZ -am
>> -amd. I
>> >> > think that we have other work (gradle support) underway that will make
>> >> this
>> >> > a non-issue since gradle automatically does even better than the
>> profile
>> >> or
>> >> > -am -amd.
>> >> >
>> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
>> >> rmannibu...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi guys,
>> >> >>
>> >> >> java/python/go/xxx support is great but as a developer you rarely
>> hack
>> >> >> on them all.
>> >> >>
>> >> >> For that reason I opened https://github.com/apache/beam/pull/4173.
>> >> >>
>> >> >> Goal is to give each developer a way to build the whole project and
>> >> >> all the code he can impact at once but without caring of the code he
>> >> >> doesn't modify at all - other languages.
>> >> >>
>> >> >> Wdyt?
>> >> >>
>> >> >> Romain Manni-Bucau
>> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >> >>
>> >>
>>

Re: [discuss] java profile

[Lukasz Cwik]

Romain, that will build the entire project. I think you want to execute
(from the root of the project):
./gradlew :beam-sdks-parent:beam-sdks-python:build

On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau 
wrote:

> gradle build --no-daemon
>
> (with gradle 4.2)
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles :
> > What is the gradle command you are using to build just the Python SDK?
> >
> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Hmm,
> >>
> >> issue is the same with gradle (locally python build takes 15mn alone
> >> which is as much as the java build and it is not parallelized I think)
> >>
> >> pl is not as smooth since it means doing it on each command whereas
> >> the proposal is automatically activated through settings.xml
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
> >>
> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
> >> > I think you can already mostly do this with mvn -pl sdks/XYZ -am
> -amd. I
> >> > think that we have other work (gradle support) underway that will make
> >> this
> >> > a non-issue since gradle automatically does even better than the
> profile
> >> or
> >> > -am -amd.
> >> >
> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
> >> rmannibu...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi guys,
> >> >>
> >> >> java/python/go/xxx support is great but as a developer you rarely
> hack
> >> >> on them all.
> >> >>
> >> >> For that reason I opened https://github.com/apache/beam/pull/4173.
> >> >>
> >> >> Goal is to give each developer a way to build the whole project and
> >> >> all the code he can impact at once but without caring of the code he
> >> >> doesn't modify at all - other languages.
> >> >>
> >> >> Wdyt?
> >> >>
> >> >> Romain Manni-Bucau
> >> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >> >>
> >>
>

Re: [discuss] java profile

[Romain Manni-Bucau]

gradle build --no-daemon

(with gradle 4.2)

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-27 20:21 GMT+01:00 Kenneth Knowles :
> What is the gradle command you are using to build just the Python SDK?
>
> On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau 
> wrote:
>
>> Hmm,
>>
>> issue is the same with gradle (locally python build takes 15mn alone
>> which is as much as the java build and it is not parallelized I think)
>>
>> pl is not as smooth since it means doing it on each command whereas
>> the proposal is automatically activated through settings.xml
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>
>>
>> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
>> > I think you can already mostly do this with mvn -pl sdks/XYZ -am -amd. I
>> > think that we have other work (gradle support) underway that will make
>> this
>> > a non-issue since gradle automatically does even better than the profile
>> or
>> > -am -amd.
>> >
>> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com>
>> > wrote:
>> >
>> >> Hi guys,
>> >>
>> >> java/python/go/xxx support is great but as a developer you rarely hack
>> >> on them all.
>> >>
>> >> For that reason I opened https://github.com/apache/beam/pull/4173.
>> >>
>> >> Goal is to give each developer a way to build the whole project and
>> >> all the code he can impact at once but without caring of the code he
>> >> doesn't modify at all - other languages.
>> >>
>> >> Wdyt?
>> >>
>> >> Romain Manni-Bucau
>> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>> >>
>>

Re: [discuss] java profile

[Kenneth Knowles]

What is the gradle command you are using to build just the Python SDK?

On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau 
wrote:

> Hmm,
>
> issue is the same with gradle (locally python build takes 15mn alone
> which is as much as the java build and it is not parallelized I think)
>
> pl is not as smooth since it means doing it on each command whereas
> the proposal is automatically activated through settings.xml
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>
>
> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
> > I think you can already mostly do this with mvn -pl sdks/XYZ -am -amd. I
> > think that we have other work (gradle support) underway that will make
> this
> > a non-issue since gradle automatically does even better than the profile
> or
> > -am -amd.
> >
> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >
> >> Hi guys,
> >>
> >> java/python/go/xxx support is great but as a developer you rarely hack
> >> on them all.
> >>
> >> For that reason I opened https://github.com/apache/beam/pull/4173.
> >>
> >> Goal is to give each developer a way to build the whole project and
> >> all the code he can impact at once but without caring of the code he
> >> doesn't modify at all - other languages.
> >>
> >> Wdyt?
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
> >>
>

Re: [discuss] java profile

[Romain Manni-Bucau]

Hmm,

issue is the same with gradle (locally python build takes 15mn alone
which is as much as the java build and it is not parallelized I think)

pl is not as smooth since it means doing it on each command whereas
the proposal is automatically activated through settings.xml

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn


2017-11-27 20:07 GMT+01:00 Kenneth Knowles :
> I think you can already mostly do this with mvn -pl sdks/XYZ -am -amd. I
> think that we have other work (gradle support) underway that will make this
> a non-issue since gradle automatically does even better than the profile or
> -am -amd.
>
> On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau 
> wrote:
>
>> Hi guys,
>>
>> java/python/go/xxx support is great but as a developer you rarely hack
>> on them all.
>>
>> For that reason I opened https://github.com/apache/beam/pull/4173.
>>
>> Goal is to give each developer a way to build the whole project and
>> all the code he can impact at once but without caring of the code he
>> doesn't modify at all - other languages.
>>
>> Wdyt?
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>>

Re: [discuss] java profile

[Kenneth Knowles]

I think you can already mostly do this with mvn -pl sdks/XYZ -am -amd. I
think that we have other work (gradle support) underway that will make this
a non-issue since gradle automatically does even better than the profile or
-am -amd.

On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau 
wrote:

> Hi guys,
>
> java/python/go/xxx support is great but as a developer you rarely hack
> on them all.
>
> For that reason I opened https://github.com/apache/beam/pull/4173.
>
> Goal is to give each developer a way to build the whole project and
> all the code he can impact at once but without caring of the code he
> doesn't modify at all - other languages.
>
> Wdyt?
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn
>