On 03/27/2015 03:20 PM, Brad King wrote: > I've drafted the attached document in Markdown to make it easy to paste > into a Github issue or comment for discussion with Ninja upstream. > > Review on our side should be about whether the proposal is sufficient
I've updated the "Automatic Module Dependencies" to split the `scan` tool execution out into its own step. The depgen file needs to be an output of a rule so it can be regenerated if it is deleted. -Brad
Ninja and Fortran ================= Ninja has excellent support for implicit dependencies of languages that use the `C` preprocessor. Fortran adds another level of complexity for build systems because its "module" system adds implicit *outputs* to compilation nodes that are referenced as implicit dependencies of other compilation nodes. These implicit outputs and dependencies may be discovered only during the build process when all (possibly generated) files are available for a given source file to be preprocessed. Below we introduce Fortran modules and their build system requirements. Then we propose new Ninja features to support them. Fortran Modules --------------- The Fortran module system is defined in section 11.2 of the [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a [Draft PDF][]. [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459 [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf Each Fortran source file may provide zero or more modules: $ cat provider.F90 MODULE mymod ... END and also consume zero or more modules: $ cat consumer.F90 USE MODULE mymod ... Compilation of a source file consuming a module requires the `.mod` file to be present: $ f95 -c consumer.F90 ... Fatal Error: Can't open module file 'mymod.mod' for reading... Compilation of a source file providing a module generates a `.mod` file for it: $ f95 -c provider.F90 $ ls *.o *.mod provider.o mymod.mod Once the module has been provided then it can be used: $ f95 -c consumer.F90 $ ls *.o *.mod consumer.o provider.o mymod.mod If the source providing a module is modified and recompiled, the module file *may or may not* be updated, depending on whether the interface exposed in the module changes. In general modules used by a source file may be found anywhere in a search path (e.g. `-I` paths). They may be provided by another source in the same build or may be found in external directories existing on the system. The only restriction on module dependencies is that they are acyclic. Manual Module Dependencies -------------------------- The above example source files may be built correctly by Ninja using manual specification of the module dependencies. We'll ignore preprocessor implicit dependencies for now: $ cat build.ninja rule FC command = f95 -o $OBJ -c $in restat = $RESTAT build provider.o mymod.mod: FC provider.F90 OBJ = provider.o RESTAT = 1 build consumer.o: FC consumer.F90 | mymod.mod OBJ = consumer.o We use the `OBJ` variable in `FC` because `$out` would contain the module as well as the object file. We use `restat` for the provider because `mymod.mod` may not be updated by the compiler. We can see that the above works in a test session: $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 [2/2] f95 -o consumer.o -c consumer.F90 $ ninja -v consumer.o ninja: no work to do. $ touch provider.F90 $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 $ ninja -v consumer.o ninja: no work to do. This example serves as a conceptual model for the basic dependency structure Fortran needs. In practice Fortran source files provide and consume many modules so maintaining the dependencies by hand is impractical. The build system must be able to discover these implicit outputs and implicit dependencies automatically. Automatic Module Dependencies ----------------------------- In order to scan for modules provided or consumed by a source file, it must first be preprocessed. Therefore we split compilation into three steps: 1. A `FP` rule to preprocess a source file: rule FP command = f95 -o $out -E $in -MMD -MT $out -MF $out.d deps = gcc depfile = $out.d This rule uses normal `depfile` for its implicit preprocessing dependencies. Preprocessing does not require the `.mod` files to exist. 2. A `FD` rule to scan for module dependency information and tell Ninja to load it using a `depgen` feature proposed below. rule FD command = scan $in $OBJ $out depgen = $out The user-provided `scan` tool will read the preprocessed source from `$in`, detect the implicit outputs and dependencies of the object file named by `$OBJ`, and write the build plan updates to the `$out` file also specified as the `depgen` file. The `depgen` file will tell Ninja, through some syntax, how to update its build plan. It will inject implicit outputs and dependencies into the compilation node for `$OBJ`. 3. A `FC` rule will compile the already-preprocessed source to produce an object file and possibly some `.mod` files. rule FC command = f95 -o $out -c $in Build statements using this rule can have an implicit or order-only dependency on the corresponding `FD` rule output to make sure implicit outputs and dependencies are known. With these rules we may use the following build statements for our above example: build provider.pp.f90: FP provider.F90 build provider.pp.f90.g: FD provider.pp.f90 OBJ = provider.o build provider.o: FC provider.pp.f90 || provider.pp.f90.g build consumer.pp.f90: FP consumer.F90 build consumer.pp.f90.g: FD consumer.pp.f90 OBJ = consumer.o build consumer.o: FC consumer.pp.f90 || consumer.pp.f90.g The `provider.pp.f90.g` file will: * Add `mymod.mod` as an *implicit output* of the `build provider.o` statement. * Mark the `build provider.o` node with the `restat` option so that consumers do not rebuild after `provider.o` if the `mymod.mod` implicit output does not change. * Add or update an *implicit build statement* of the form: build mymod.mod.proxy: phony mymod.mod The `consumer.pp.f90.g` file will: * Add or update an *implicit build statement* of the form: build mymod.mod.proxy: phony fallback * Add `mymod.mod.proxy` as an *implicit dependency* of the `build consumer.o` statement. The phony `mymod.mod.proxy` implicit build statement is needed because the `scan` tool reading sources that consume `mymod` does not know where the `mymod.mod` file will be located on disk or even if it is provided by another source in the build: * Compilations consuming `mymod.mod` get an implicit dependency on `mymod.mod.proxy` for which an implicit build statement is always provided. * The compilation providing `mymod.mod`, if any, knows where the file will be located on disk. The path is added as a dependency of `mymod.mod.proxy` so that the phony target will be out of date when `mymod.mod` is modified and the consumers will recompile. * The `fallback` file is created once by the `scan` tool and never updated. It satisfies proxy dependencies for external (system) modules not provided by another source file in the build. In such cases no modification to sources within the build would cause the proxy to become out of date. Ninja Features Needed --------------------- The above approach requires Ninja to be taught some new features. In particular: 1. Add a notion of *implicit outputs* on build plan nodes that are not part of `$out` but that Ninja knows are generated. 2. Teach Ninja to recognize the `depgen` option on a rule and load the named file immediately after an instance of the rule is brought up to date (whether or not the instance runs). The `depgen` file, through some syntax, will specify updates to existing build nodes and/or add new build nodes: * Add a new build statement (at least phony rules). * Add implicit dependencies to an existing build statement. * Add implicit outputs to an existing build statement. * Enable `restat` on an existing build statement. We have not yet designed a specific syntax for the `depgen` file and are open to suggestions. It should be a format that is easy to generate, fast to parse, and extensible beyond the above operations in the future.
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
