On 10 Jul 2014, at 9:19 pm, Daniel Lacasse <daniel.lacass...@gmail.com> wrote:
> If you are depending on a big project, let say Boost, you get a big > performance hit. Please correct me if I'm wrong. > > Story: Better input selection for AbstractNativeCompileTask > The AbstractNativeCompileTask specified @InputFiles on FileCollection > includes. The up-to-date check would then hash every files in all include > path. This isn’t quite what happens. It doesn’t hash every header file on every build invocation - it hashes each header file once and caches the result, and then uses (last modified timestamp, length) to invalidate this cached hash. There are really two problems here: 1. Gradle considers files that aren’t actually being used as inputs (and on the flip side, is ignoring other files that are). 2. The implementation of the up-to-date checks are inefficient. More on this below. > In practice, this doesn't work. You may only be using a couple header files > directly and because of the header dependencies, this may only gets to a > handful of files. Boost has about 10k of header files which is the perfect > example for showing this problem. > > Implementation > The proposed solution is to have a list of include directory as the Input for > this task. Probably a List<File> and annotate with @Input. We would keep the > power of having File object but remove the hashing of every single files > within the include directory. I think a better option would be to have the task declare which header files each source file depends on, and have Gradle do the up-to-date checks based on this. The task calculates this stuff anyway. > > Story: Improve incremental compilation > Since we will only rely on header parsing for header dependencies, it would > be a good idea to improve the current processing. I don't take credit this > next improvement. For the full explanation, see > http://martine.github.io/ninja/manual.html#ref_headers. Basically, instead of > parsing all files in order to find the #include which is the current Gradle > way, we can use the initial compiler run with specific compiler option to > output the dependency graph as the compiler sees it. In gcc and clang it's > the -MMD and -MF flags and msvc it's the /showIncludes flag. > > Implementation: > For msvc, a new option is needed on the toolchain for telling Gradle what is > the prefix String for each header dependency output line from cl.exe. This > option is needed for non-english Visual Studio installation. This can be > auto-detected by Gradle but just in case Microsoft decide to port Visual > Studio in English pirate. For gcc and clang, -MF would point to a file in the > build/tmp/<taskName> directory. > > For improved speed - and I assume that Gradle cache is extremely fast and > efficient - Gradle would save the header dependency information in it's > cache. This means Gradle would parse the gcc and clang dependency file as > soon as it's generated. > > The rest of the incremental compile stays the same. I don’t want to make this change. Firstly, I don’t think it’s inherently faster, and secondly, it prevents us doing interesting performance improvements later. Instead, we should make the existing implementation faster, in the following ways: 1. Use a single cache of parsed header files that is shared by (at least) all compilation tasks in a build, or (better) all compilation tasks on a given machine. 2. Calculate whether a the graph for a given (header file, include roots) is up-to-date or not once per build, rather than once per (file, task). 3. Change the daemon to watch for file system changes, and invalidate the above stuff as things change. 4. Make the up-to-date check faster for those files we know will not change (or are very unlikely to change), such as files in the Gradle artefact cache, or those managed by the system package manager. There would have to be some way to tell Gradle that you want or don’t want this optimisation. -- Adam Murdoch Gradle Co-founder http://www.gradle.org CTO Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
