One pitfall I noted with javac wrappers is that if you dont clean - and loose javac incremental support then deleted classes stay here and are considered in the output. Common example is a nested class which has been deleted leading to a corrupted enclosing class or a test which is ran but deleted from java/. Any known way to protect us from it and keep the uncremental support for big modules?
Le 25 janv. 2018 01:22, "Lukasz Cwik" <lc...@google.com> a écrit : > Dependency driven works, incremental works for most java modules. > I use incremental almost all the time and just do one validation pass at > the end before opening the PR where I use '--rerun-tasks' to be sure. > Allows me to iterate on a task in seconds. > > On Wed, Jan 24, 2018 at 4:07 PM, Kenneth Knowles <k...@google.com> wrote: > >> These are two different things: dependency-driven build (which works) and >> incremental build (which seems not to, at least right now?). >> >> On Wed, Jan 24, 2018 at 2:24 PM, Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> Hmm, I'll try to refine it then next time we work with Ismael but can be >>> a setup issue or a human (bad command) issue at the end. Thanks for the >>> help, will make next iteration way easier probably :) >>> >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> <https://rmannibucau.metawerx.net/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> >>> >>> 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>> >>>> Tasks always run any dependencies that are required. So if you ask to >>>> run test it shouldn't run javadoc/checkstyle/... but should compile the >>>> code and compile the code of all dependencies. test should never have a >>>> dependency on checkstyle or javadoc or similar 'check' like tasks as they >>>> shouldn't be needed. >>>> >>>> I set up the gradle build so that everytime you run a command in >>>> gradle, it generates a task dependency tree dot file (look for visteg.dot >>>> inside build/reports). I uploaded this one to imgur[1] for the >>>> ':sdks:java:core:build' task to show what tasks are required. Note that >>>> 'sdks:java:core:test' doesn't depend on checkstyle or spotless. >>>> >>>> 1: https://imgur.com/a/ZvYUX >>>> >>>> On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> Hmm, do I miss something or it only works for iterative runs when >>>>> trying to identify an issue and not for the case you rebuild due to code >>>>> changes (where you would need like 5-6 tasks at least - generate, compile, >>>>> test, ...)? >>>>> >>>>> In case it is unclear: there are 2 needs: direct execution/task -> >>>>> fulfilled and clarified now (just a doc issue I think), fast cycle >>>>> skipping >>>>> not mandatory tasks like style related ones >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> >>>>> >>>>> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>> >>>>>> Gradle already has each task explicitly broken out. Kenn is pointing >>>>>> out that you your development use case shouldn't use the './gradlew >>>>>> :sdks:java:core:build' task since it is really an aggregator that >>>>>> represents do everything within that project. This is the current list of >>>>>> tasks available for :sdks:java:core: >>>>>> :sdks:java:core:assemble - Assembles the outputs of this project. >>>>>> :sdks:java:core:build - Assembles and tests this project. >>>>>> :sdks:java:core:buildDependents - Assembles and tests this project >>>>>> and all projects that depend on it. >>>>>> :sdks:java:core:buildEnvironment - Displays all buildscript >>>>>> dependencies declared in project :sdks:java:core. >>>>>> :sdks:java:core:buildNeeded - Assembles and tests this project and >>>>>> all projects it depends on. >>>>>> :sdks:java:core:check - Runs all checks. >>>>>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>>>>> classes >>>>>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>>>>> classes >>>>>> :sdks:java:core:classes - Assembles main classes. >>>>>> :sdks:java:core:clean - Deletes the build directory. >>>>>> :sdks:java:core:compileJava - Compiles main Java source. >>>>>> :sdks:java:core:compileTestJava - Compiles test Java source. >>>>>> :sdks:java:core:components - Displays the components produced by >>>>>> project :sdks:java:core. [incubating] >>>>>> :sdks:java:core:dependencies - Displays all dependencies declared in >>>>>> project :sdks:java:core. >>>>>> :sdks:java:core:dependencyInsight - Displays the insight into a >>>>>> specific dependency in project :sdks:java:core. >>>>>> :sdks:java:core:dependencyReport - Generates a report about your >>>>>> library dependencies. >>>>>> :sdks:java:core:dependentComponents - Displays the dependent >>>>>> components of components in project :sdks:java:core. [incubating] >>>>>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes >>>>>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes >>>>>> :sdks:java:core:generateAvroJava - Generates main Avro Java source >>>>>> files from schema/protocol definition files. >>>>>> :sdks:java:core:generateAvroProtocol - Generates main Avro protocol >>>>>> definition files from IDL files. >>>>>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java >>>>>> source files from schema/protocol definition files. >>>>>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>>>>> protocol definition files from IDL files. >>>>>> :sdks:java:core:help - Displays a help message. >>>>>> :sdks:java:core:htmlDependencyReport - Generates an HTML report >>>>>> about your library dependencies. >>>>>> :sdks:java:core:install - Installs the archives artifacts into the >>>>>> local Maven repository. >>>>>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>>>>> coverage metrics based on specified rules for the test task. >>>>>> :sdks:java:core:jacocoTestReport - Generates code coverage report >>>>>> for the test task. >>>>>> :sdks:java:core:jar - Assembles a jar archive containing the main >>>>>> classes. >>>>>> :sdks:java:core:javadoc - Generates Javadoc API documentation for >>>>>> the main source code. >>>>>> :sdks:java:core:knows - Do you know who knows? >>>>>> :sdks:java:core:model - Displays the configuration model of project >>>>>> :sdks:java:core. [incubating] >>>>>> :sdks:java:core:packageTests - >>>>>> :sdks:java:core:processResources - Processes main resources. >>>>>> :sdks:java:core:processTestResources - Processes test resources. >>>>>> :sdks:java:core:projectReport - Generates a report about your >>>>>> project. >>>>>> :sdks:java:core:projects - Displays the sub-projects of project >>>>>> :sdks:java:core. >>>>>> :sdks:java:core:properties - Displays the properties of project >>>>>> :sdks:java:core. >>>>>> :sdks:java:core:propertyReport - Generates a report about your >>>>>> properties. >>>>>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>>>>> runtime dependencies >>>>>> :sdks:java:core:shadowTestJar - >>>>>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>>>>> sourcecode in-place. >>>>>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>>>>> formatting steps. >>>>>> :sdks:java:core:spotlessJava - >>>>>> :sdks:java:core:spotlessJavaApply - >>>>>> :sdks:java:core:spotlessJavaCheck - >>>>>> :sdks:java:core:taskReport - Generates a report about your tasks. >>>>>> :sdks:java:core:tasks - Displays the tasks runnable from project >>>>>> :sdks:java:core. >>>>>> :sdks:java:core:testClasses - Assembles test classes. >>>>>> :sdks:java:core:test - Runs the unit tests. >>>>>> :sdks:java:core:updateOfflineRepository - >>>>>> >>>>>> So if you specify the ':sdks:java:core:test' task then only the >>>>>> tests run (and any dependent tasks which are not up to date), while >>>>>> ':sdks:java:core:check' >>>>>> runs the suite of all checks. If your working on two modules and want to >>>>>> run the tests for both you really want to specify each explicit end goal >>>>>> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >>>>>> Unfortunately incremental builds (https://issues.apache.org/jir >>>>>> a/browse/BEAM-3253) are unreliable in a few of the modules (like sql >>>>>> and python) so you may find that you need to specify --rerun-tasks to >>>>>> make >>>>>> sure that all tasks are rerun even if Gradle thinks they are up to date. >>>>>> >>>>>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com> wrote: >>>>>> >>>>>>> Can have mischecked gradle setup but I don't think we are here yet, >>>>>>> if you are not bound in a module and work accross 2 modules and iterate >>>>>>> between working on both and one, you will likely not bypass the "checks" >>>>>>> in a satisfying way without a long -x command, is there a magic flag I >>>>>>> missed? >>>>>>> Also not sure about the last point and how gradle helps here - it is >>>>>>> rather the opposite due to the way it loads it model IMHO - so not sure >>>>>>> what would be the consequence in terms of action(s) but can have missed >>>>>>> the >>>>>>> point. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Romain Manni-Bucau >>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>>> >>>>>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>>>> >>>>>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>> >>>>>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>>>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>>>>> >>>>>>>> >>>>>>>> Yes, I read it right. We all want the same thing - not doing a >>>>>>>> bunch of extra useless unrequested stuff when developing. The concept >>>>>>>> of >>>>>>>> skipping is backwards. We don't need configs that skip things, because >>>>>>>> in a >>>>>>>> correct dependency-driven build they are already not running. >>>>>>>> >>>>>>>> So since I don't want to pretend to know gradle's invocations yet I >>>>>>>> will call it $TOOL. Here's a collection of imaginary commands: >>>>>>>> >>>>>>>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core >>>>>>>> or whatever >>>>>>>> $TOOL :sdks:java:core:findbugs >>>>>>>> $TOOL :sdks:java:core:checkstyle >>>>>>>> $TOOL :sdks:java:core:javadoc >>>>>>>> >>>>>>>> None of these causes any of the others to run. Anything else is a >>>>>>>> bug. The `findbugs` and `test` cause a build of the needed jars and >>>>>>>> nothing >>>>>>>> else. >>>>>>>> >>>>>>>> Another example: >>>>>>>> >>>>>>>> $TOOL :runners:core-java:unittest >>>>>>>> >>>>>>>> This builds the model, the core SDK, and the runners-core module, >>>>>>>> then runs the unit tests of the runners-core module. It does not test >>>>>>>> SDK >>>>>>>> core, or run any javadoc, findbugs, or checkstyle on any module. >>>>>>>> Anything >>>>>>>> else is a bug. >>>>>>>> >>>>>>>> Now, to build a precommit that is easy to reproduce on one line, >>>>>>>> you could build a compound task >>>>>>>> >>>>>>>> $TOOL :sdks:java:core:precommit # runs a selection of targets >>>>>>>> that we define >>>>>>>> >>>>>>>> At this point you might want to skip things from the :verify task >>>>>>>> here. But really, you probably just want to run the things you are >>>>>>>> interested in and you don't need custom hooks in the aggregated task. >>>>>>>> >>>>>>>> My understanding is that gradle can support all of this, if we are >>>>>>>> disciplined. Getting to this point is the main/only reason I supported >>>>>>>> gradle. >>>>>>>> >>>>>>>> Kenn >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> Kenn >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >> >>>>>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>>>>> b/examples/java/build.gradle >>>>>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>>>>> >> ] >>>>>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>>>>> description: "Run tests >>>>>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>>>>> >> "--project=apache-beam-testing", >>>>>>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> Romain Manni-Bucau >>>>>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>>>>> https://github.com/rmannibucau> | >>>>>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>>>>> >> >>>>>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré < >>>>>>>>>>> j...@nanthrax.net >>>>>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>>>>> >> >>>>>>>>>>> >> Hi Romain, >>>>>>>>>>> >> >>>>>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>>>>> tasks & projects names. >>>>>>>>>>> >> >>>>>>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>>>>>> >> >>>>>>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>>>>>> gradle projects >>>>>>>>>>> >> provides the list of projects/modules. >>>>>>>>>>> >> >>>>>>>>>>> >> Regards >>>>>>>>>>> >> JB >>>>>>>>>>> >> >>>>>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause they >>>>>>>>>>> are easily outdated and >>>>>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>>>>> renaming properly the tasks >>>>>>>>>>> >> > and maybe writing what is done in build files - since >>>>>>>>>>> it is code and even "api >>>>>>>>>>> >> > for dev", it requires as much comments than the main >>>>>>>>>>> api - can be better to start. >>>>>>>>>>> >> > >>>>>>>>>>> >> > Also a big switch flag to bypass >>>>>>>>>>> checkstyle/findbugs/... can be good while in >>>>>>>>>>> >> > dev since these phases cost a looot for nothing while >>>>>>>>>>> you validates your code in >>>>>>>>>>> >> > runners modules for instance. >>>>>>>>>>> >> > >>>>>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" < >>>>>>>>>>> k...@google.com <mailto:k...@google.com> >>>>>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a >>>>>>>>>>> écrit : >>>>>>>>>>> >> > >>>>>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain >>>>>>>>>>> Manni-Bucau <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com >>>>>>>>>>> > >>>>>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>>>>> >> > >>>>>>>>>>> >> > @Kenneth: why not dropping the doc for a script >>>>>>>>>>> with comments in the >>>>>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > That's cool, too, but also any shell one liner can >>>>>>>>>>> be a gradle one >>>>>>>>>>> >> liner or >>>>>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>>>>> command that you cannot >>>>>>>>>>> >> > guess easily for a different one that you still >>>>>>>>>>> can't guess easily. >>>>>>>>>>> >> > >>>>>>>>>>> >> > For example, are the SparkRunner ValidatesRunner >>>>>>>>>>> tests in the >>>>>>>>>>> >> SparkRunner or >>>>>>>>>>> >> > the core SDK or a third module that integrates the >>>>>>>>>>> two? And why would you >>>>>>>>>>> >> > know that the example ITs are called >>>>>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>>>>> >> doesn't even >>>>>>>>>>> >> > make sense really to have "precommit" or >>>>>>>>>>> "postcommit" except as aliases to >>>>>>>>>>> >> > make it easy to repro Jenkins' behavior - they have >>>>>>>>>>> no other intrinsic >>>>>>>>>>> >> meaning. >>>>>>>>>>> >> > >>>>>>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>>>>>> description" to one >>>>>>>>>>> >> liner >>>>>>>>>>> >> > to help people navigate the targets that we set up. >>>>>>>>>>> Some web page or doc >>>>>>>>>>> >> > that people can just quickly scan to find out to do >>>>>>>>>>> common things, easier >>>>>>>>>>> >> > than groovy or XML. >>>>>>>>>>> >> > >>>>>>>>>>> >> > Kenn >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> >>>>>>>>>>> >> -- >>>>>>>>>>> >> Jean-Baptiste Onofré >>>>>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>>>>> >> http://blog.nanthrax.net >>>>>>>>>>> >> Talend - http://www.talend.com >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> > >>>>>>>>>>> > -- >>>>>>>>>>> > Jean-Baptiste Onofré >>>>>>>>>>> > jbono...@apache.org >>>>>>>>>>> > http://blog.nanthrax.net >>>>>>>>>>> > Talend - http://www.talend.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
