Le 29 nov. 2017 01:29, "Robert Bradshaw" <rober...@google.com> a écrit :
I am curious what you mean by the "Python Build" as by nature there's no (significant) compilation that happens. Python tests, on the other hands, are completely sequential (and very redundant--each test file is run at least three times). With the potential of Java builds becoming fast enough that Python becomes the long (or at least a significant) factor, I'm staring to get motivated to look at this more closely :). The python generated and executed code. I didnt look more than that bit it looks like setup (pip) and tests are not that fast even if several are skipped (?? A missing dep shouldnt reduce the coverage right?) With respect to correct incremental builds, not only is it possible, but it can be done at scale (e.g. http://google-engtools.blogspot.com/2011/08/build-in- cloud-how-build-system-works.html https://blog.bazel.build/2017/08/25/introducing-sandboxfs.html https://nixos.org/nix/) One trick is to disallow tools to even look at inputs that are not declared. As you've probably guessed, I'm partial to reproducible builds (and reproducible research in general) myself. Unfortunately the explorations into bazel seem to indicate that the third-party dependency system isn't sufficiently mature (vs. the one-codebase environment it was developed in). It is a theory yes but in practise for a multimodule project you have to not respect it. It only impacts you with indirect code path like SPI or reflection. Beam does the first one at least. Dont get me wrong, it is not a blocker but it is also not magic and you will need to add a no daemon and clean build to avoid caching issues it is easy to have with gradle in dev mode. With regards to the original topic, hopefully the build system can just take care of testing exactly what needs to be tested based on changes. Note that this will likely cross language barriers more and more as the portability work progresses (e.g. we'll have Python-on-Flink tests that should be run if either the Python SDK or the Flink runner gets modified). The git integration of maven we talked earlier does that. I guess/hope gradle has the same but it is bound to the scm rather than the build tool. Worse case a git (status based) script should enable to launch that. - Robert On Tue, Nov 28, 2017 at 12:49 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Lukasz: only for an isolated "system" which is a module - assuming you still > want to be able to build a submodule without building and revalidating the > whole tree which is important in dev IMO. This means you shouldnt handle > inputs outside "current" module and therefore miss easily some (typically > test) execution/coverage - incremental compilation/style check etc is not > that costly compared to tests. So this is far to be as straight forward as > it looks if you want to keep the same guarantees. Once again for > checkstyle/findbugs/javac/rat etc this is trivial - and even with maven > actually - but doesnt save a significant amount of time in dev whereas tests > do. > > > Le 28 nov. 2017 20:11, "Kenneth Knowles" <k...@google.com> a écrit : > > I seem to remember a tool called `make` that was pretty good at this. > > On Tue, Nov 28, 2017 at 10:47 AM, Lukasz Cwik <lc...@google.com> wrote: >> >> Its been well shown that a build system that uses input/output set change >> detection can correctly implement incremental builds. Build systems are not >> tied to knowing the internal details of how Java compiles things. Knowing >> that there are some inputs, a process, and some outputs is enough to know >> when the process needs to be rerun. >> >> On Mon, Nov 27, 2017 at 9:53 PM, Romain Manni-Bucau >> <rmannibu...@gmail.com> wrote: >>> >>> Hmm, no. >>> >>> Incremental build is never correctly implemented cause there is just no >>> way to detect some dependencies statically with java code - or any dynamic >>> language. >>> >>> Side note: same applies for gradle daemon usage BTW. >>> >>> After if the list is not maintained it is a bug at the same level than >>> coding a toString() with "null.toString()". This is not very hard to handle >>> the list of modules and worse case a mvnextension can make it coded if you >>> feel more comfortable with this kind of solution. >>> >>> Le 27 nov. 2017 23:12, "Lukasz Cwik" <lc...@google.com.invalid> a écrit : >>>> >>>> Manually whitelisting/blacklisting sub-modules is error prone since it >>>> hides issues due to incorrectly maintaining that list is the same >>>> argument >>>> as if the build process doesn't correctly invoke an incremental build >>>> process. >>>> >>>> On Mon, Nov 27, 2017 at 1:45 PM, Romain Manni-Bucau >>>> <rmannibu...@gmail.com> >>>> wrote: >>>> >>>> > Well for validation builds- pre PR, incremental support is pointless >>>> > since >>>> > it easily hides issues die to caching so a solution saving half of the >>>> > build without loosing anuyhing would still be good IMHO. >>>> > >>>> > Le 27 nov. 2017 21:12, "Lukasz Cwik" <lc...@google.com.invalid> a >>>> > écrit : >>>> > >>>> > > Incremental builds aren't correctly setup right now so your likely >>>> > > to see >>>> > > Python/Go rebuild even if there were no changes. See >>>> > > https://issues.apache.org/jira/browse/BEAM-3253 >>>> > > >>>> > > On Mon, Nov 27, 2017 at 11:46 AM, Romain Manni-Bucau < >>>> > > rmannibu...@gmail.com> >>>> > > wrote: >>>> > > >>>> > > > that was the goal: validate there was no side effect of the >>>> > > > changes on >>>> > > > the whole project. Now the "not java" part of the build will not >>>> > > > be >>>> > > > impacted by java changed so this is the part i want to skip since >>>> > > > it >>>> > > > takes a lot of time and I have guarantees it is safe to skip them. >>>> > > > >>>> > > > Romain Manni-Bucau >>>> > > > @rmannibucau | Blog | Old Blog | Github | LinkedIn >>>> > > > >>>> > > > >>>> > > > 2017-11-27 20:28 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid >: >>>> > > > > Romain, that will build the entire project. I think you want to >>>> > execute >>>> > > > > (from the root of the project): >>>> > > > > ./gradlew :beam-sdks-parent:beam-sdks-python:build >>>> > > > > >>>> > > > > On Mon, Nov 27, 2017 at 11:25 AM, Romain Manni-Bucau < >>>> > > > rmannibu...@gmail.com> >>>> > > > > wrote: >>>> > > > > >>>> > > > >> gradle build --no-daemon >>>> > > > >> >>>> > > > >> (with gradle 4.2) >>>> > > > >> >>>> > > > >> Romain Manni-Bucau >>>> > > > >> @rmannibucau | Blog | Old Blog | Github | LinkedIn >>>> > > > >> >>>> > > > >> >>>> > > > >> 2017-11-27 20:21 GMT+01:00 Kenneth Knowles >>>> > > > >> <k...@google.com.invalid >>>> > >: >>>> > > > >> > What is the gradle command you are using to build just the >>>> > > > >> > Python >>>> > > SDK? >>>> > > > >> > >>>> > > > >> > On Mon, Nov 27, 2017 at 11:19 AM, Romain Manni-Bucau < >>>> > > > >> rmannibu...@gmail.com> >>>> > > > >> > wrote: >>>> > > > >> > >>>> > > > >> >> Hmm, >>>> > > > >> >> >>>> > > > >> >> issue is the same with gradle (locally python build takes >>>> > > > >> >> 15mn >>>> > > alone >>>> > > > >> >> which is as much as the java build and it is not >>>> > > > >> >> parallelized I >>>> > > > think) >>>> > > > >> >> >>>> > > > >> >> pl is not as smooth since it means doing it on each command >>>> > whereas >>>> > > > >> >> the proposal is automatically activated through settings.xml >>>> > > > >> >> >>>> > > > >> >> Romain Manni-Bucau >>>> > > > >> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn >>>> > > > >> >> >>>> > > > >> >> >>>> > > > >> >> 2017-11-27 20:07 GMT+01:00 Kenneth Knowles >>>> > <k...@google.com.invalid >>>> > > >: >>>> > > > >> >> > I think you can already mostly do this with mvn -pl >>>> > > > >> >> > sdks/XYZ >>>> > -am >>>> > > > >> -amd. I >>>> > > > >> >> > think that we have other work (gradle support) underway >>>> > > > >> >> > that >>>> > will >>>> > > > make >>>> > > > >> >> this >>>> > > > >> >> > a non-issue since gradle automatically does even better >>>> > > > >> >> > than >>>> > the >>>> > > > >> profile >>>> > > > >> >> or >>>> > > > >> >> > -am -amd. >>>> > > > >> >> > >>>> > > > >> >> > On Mon, Nov 27, 2017 at 11:01 AM, Romain Manni-Bucau < >>>> > > > >> >> rmannibu...@gmail.com> >>>> > > > >> >> > wrote: >>>> > > > >> >> > >>>> > > > >> >> >> Hi guys, >>>> > > > >> >> >> >>>> > > > >> >> >> java/python/go/xxx support is great but as a developer >>>> > > > >> >> >> you >>>> > > rarely >>>> > > > >> hack >>>> > > > >> >> >> on them all. >>>> > > > >> >> >> >>>> > > > >> >> >> For that reason I opened https://github.com/apache/ >>>> > > beam/pull/4173 >>>> > > > . >>>> > > > >> >> >> >>>> > > > >> >> >> Goal is to give each developer a way to build the whole >>>> > project >>>> > > > and >>>> > > > >> >> >> all the code he can impact at once but without caring of >>>> > > > >> >> >> the >>>> > > code >>>> > > > he >>>> > > > >> >> >> doesn't modify at all - other languages. >>>> > > > >> >> >> >>>> > > > >> >> >> Wdyt? >>>> > > > >> >> >> >>>> > > > >> >> >> Romain Manni-Bucau >>>> > > > >> >> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn >>>> > > > >> >> >> >>>> > > > >> >> >>>> > > > >> >>>> > > > >>>> > > >>>> > >> >> > >
