On 09/04/2013, at 11:55 AM, Luke Daley <[email protected]> wrote:
> > On 27/03/2013, at 3:43 PM, Daz DeBoer <[email protected]> wrote: > >> G'day >> >> Now in master is a pretty cool new feature: you can now implement an >> 'incremental' task that is informed about exactly which input files >> have changed when the task is out of date. >> This is very useful for something like a C++ compile task, as it means >> that only the changed files need to be recompiled, rather than the >> entire set of inputs. >> >> I've got a 'draft' DSL functioning, and would appreciate any feedback >> you guys have. Here's a sample: >> >> class IncrementalSync extends DefaultTask { >> @InputFiles >> def FileCollection src >> >> @OutputDirectory >> def File destination >> >> @TaskAction >> void execute(TaskInputChanges inputs) { >> if (inputs.allOutOfDate) { >> FileUtils.forceDelete(destination) >> } >> >> inputs.outOfDate({ >> FileUtils.copyFile(change.file, targetFile(change.file)) >> } as Action) >> .removed({ >> FileUtils.forceDelete(targetFile(change.file)) >> } as Action) >> .process() >> } >> >> def targetFile(def inputFile) { >> new File(destination, change.file.name) >> } >> } > > Thinking more about this, I don't think what we have is right. I think we can > come up with a simpler API. > > interface TaskExecutionContext { > > boolean isHasPriorOutputs(); // this is an incremental execution > > void eachInputFileChange(Action<InputFileChange> action); > > interface InputFileChange { > boolean isAdded(); > boolean isModified(); > boolean isRemoved(); > File getFile(); > } > > } > > @TaskAction > void execute(TaskExecutionContext executionContext) { > if (executionContext.hasPriorOutputs) { > executionContext.eachInputFileChange { > > } > } else { > // full rebuild > } > } > > Some issues I have with the existing design: > > * I'm not sure "outOfDate" is the right term here. It doesn't quite capture > it and can imply more than what we actually mean. You could say that an > output is out of date, but it doesn't quite work for inputs. > * why do we need the outOfDate() and removed() methods? After reading the > Javadoc I'm not sure how I would use this API. What's the difference between > out-of-date and removed? I could guess, but it's not clear. > * I think we'll want to provide information that is not strictly an aspect of > the inputs or outputs, but of something emergent from the input/output > concept. That's why I think modelling an encompassing type is better. Sorry, accidentally hit send… > What I think we are trying to do is make it easy for the user to decide > whether they should do a fine grained incremental execution or a coarse > grained full. The existing isAllOutOfDate() method goes too far, based on its Javadoc: /** * Specifies if incremental build is not possible due to changed Input Properties, Output Files, etc. * In this case, every file will be considered to be 'out-of-date'. */ boolean isAllOutOfDate(); It is too presumptuous of us to say that all outputs are invalid just because an input property changed. The task author may know better and can only rebuild what is effected by the input property change. We can tell if a full build is needed with relative confidence based on the lack of task history, or by verifying that the declared outputs don't exist (if that's fast enough), hence the isHasPriorOutputs() method. It would be nice to be able to tell the task that all of the input files have changed in some way, but this seems like it would be very expensive to compute. > > >> >> Notes: >> 1. The way to implement an incremental task is to add a >> TaskInputChanges parameter to your @TaskAction method. This must be a >> typed parameter, and currently TaskInputChanges is the only parameter >> type we support (but there are plans to add more, like >> TaskOutputChanges). The reason for using a typed parameter is that >> this is the way the task tells us what it wants: I thought about an >> annotated parameter, but it seems kind of pointless when the >> annotation would imply the type anyway. (Perhaps we can add an >> annotation-based marker at a later stage, if it helps). >> >> 2. There are 2 discrete ways we report incremental changes: >> - If the _only_ change to the task execution state is changed input >> files, then TaskInputChanges.allOutOfDate() will be false, and only >> the added/changed/removed files will be notified to the >> TaskInputChanges.outOfDate() and .removed() actions. >> - In the case of non-file changes to task inputs (properties, task >> class) and changes to task output files, then Gradle will consider all >> input files to be out of date. In this case, >> TaskInputChanges.allOutOfDate() will be true, and every input file >> will be reported to the TaskInputChanges.outOfDate() action. >> >> 4. The reason for the chained action methods combined with a final >> process() method is that this allows us to stream changed inputs in >> any order, and does not require us to persist these changes for a >> subsequent method call. This is a little awkward, but doesn't force us >> to jump through hoops. We could implement a more discrete API on top, >> but it may be less efficient. >> >> 5. I haven't yet got any DSL magic applied to the TaskInputChanges >> instance, so using a closure directly isn't (yet) possible. Not sure >> how important that is for this DSL, or how tricky it will be to add. >> >> You can read more about the plans here: >> https://github.com/gradle/gradle/blob/master/design-docs/incremental-build.md >> Next steps for incremental tasks include providing access to changed >> outputs and properties (in the case a task can handle these more >> efficiently), automatically cleaning up stale outputs, and fixing some >> bugs around the incremental nature of Copy tasks (and others). >> >> -- >> Darrell (Daz) DeBoer >> Principal Engineer, Gradleware >> http://www.gradleware.com >> Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, >> CA: http://www.gradlesummit.com >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > -- Luke Daley Principal Engineer, Gradleware http://gradleware.com Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
