Adam, On Tue, 2011-08-02 at 10:14 +1000, Adam Murdoch wrote:
I may be stating the obvious and known . . .
Immediate reaction to this is that SCons and Waf have already solved
most, if not all, of the algorithmic issues relating to the header
transitivity involved here. Whilst it may be that in the Gradle context
new "from scratch" algorithms are needed, certainly implementations will
be, there is no point in ignoring the work already done.
It may also be worth looking at the Go build scenario. With goinstall,
gb, gofr, the Go people have taken convention over configuration to the
extreme in that build requires no configuration files at all. They also
have transitive import/include issues, and separate compile and link
issues.
Regarding versioning of libraries, there are POSIX conventions for
versioning of dynamic libraries (sonames) which have to be taken into
account. Windows DLLs are a whole other hell that I have no knowledge
about.
With systems such as Debian, Ubuntu, Fedora, etc. there are packaged
versions of everything and these should be used unless cross-compiling.
> One of the goals for our C++ support is to offer dependency management
> for binary artifacts built from C/C++.
>
>
> Ideally, we want to reuse the existing dependency dsl, rather than add
> a new native-specific dsl. So, you'd do something like:
>
>
> apply plugin: 'cpp'
> apply plugin: 'native-executable'
>
>
> dependencies {
> compile 'someGroup:someProject:1.2'
> }
Where is this dependency going to be resolved? Unlike JVM/Maven there
is no central repository of versioned libraries.
Also is this a dynamic or a static dependency?
The Go folk are ignoring dynamic binding, and build only static
executables. C, C++, Fortran, D on the other hand prefer dynamic
binding over static binding.
>
> This would imply that:
> * The headers for the library are downloaded, and the appropriate -I
> flag add to the compiler invocation.
Downloaded from where? What about stuff already installed? Surely
search /usr/include first before downloading anything.
Header files are rarely versioned for C, C++, Fortran, D. Go handles
things by nigh on requiring everything to be in Git, Mercurial or
Bazaar. They then provide all the versioning.
> * The library binary for the variant we are building is downloaded,
> and added to the linker invocation.
From where?
> * The dependency meta-data is included in the published dependency
> descriptor for the binary we are building.
What meta-data? Is the intention to create the equivalent of Mave but
for C, C++, Fortran, D
> * Depending on its type, the library binary is made available at
> runtime when running the tests, or building a distribution, or
> installing the binary.
Which platform? Dynamic binding or static binding? What about
different versions of different platforms?
> There is also some transitive dependency management needed:
> * Library headers may require headers from some of its dependencies to
> be available when compiling.
> * Library binaries require its runtime dependencies to be available at
> runtime.
See above :-(
> There are several aspects where dependency management for native
> binaries is different to what we current offer for the jvm:
>
>
> * Different types of artifacts are required at different points in the
> build: headers at compile time vs binaries at link time. It would be
> nice to also solve this for building jvm based projects, so that a
> project can, for example, publish an api jar for use when compiling,
> and an impl jar for use at runtime.
>
>
> * Related to this, different sets of transitive dependencies are
> required at different points in the build: header dependencies at
> compile time, nothing at link time, and full transitive closure of
> runtime dependencies at runtime. But this is also true for jvm
> projects, we just ignore the problem at the moment.
>
>
> * There are multiple variants of a given artifact, and this is a
> multi-dimensional: os, arch, debug, compiler, and so on. Again, it
> would be nice to also solve this for jvm projects, so that a project
> can, for example, publish groovy 1.7 and groovy 1.8 variants, or scala
> 2.7 and scala 2.8 variants. Or a java project with some jni native
> pieces can publish variants for the different platforms it supports.
> Or a web app project can publish fat and skinny variants. Or dev, qa
> and prod variants.
When it comes to native executables you have to depend on a specific
version of a specific platform. So for example Ubuntu 11.04 is
incompatible with Ubuntu 11.11, which is very incompatible with Fedora
15 or Windows 2000, XP, Vista, 7, etc. Then there are the continuously
updated platforms such as MacPorts (source based) and Debian Testing
(except during a freeze)
Static compilation aids somewhat, hence Gos position on linking
> This also implies that for a given project, some artifacts will be
> variant-specific (binaries, debug symbol tables), and some artifacts
> will be variant independent (headers, source files, documentation).
Which is why MacPorts and the whole of Gentoo is source based -- you
never ship executables you only ship source and compile for your system.
Fine for owners of one computer something of a disaster for people
managing hundreds or even thousands.
>
> * For a given dependency, there are potentially multiple compatible
> artifacts. For example, on linux, we can link against either a debug
> or non-debug variant. So, if we're building a debug variant,
> dependency resolution should prefer the debug variant of a dependency,
> but can fall back to a non-debug variant if that's the only one
> available. But, on windows, for example, we can't do this
> substitution.
>
>
> * Some tools are sensitive to the naming scheme we use for artifacts.
> For the java tools, we pass the path to the artifact in the cache
> directly to the tool, and this works fine pretty much regardless of
> the artifact name. However, for example, we must not include the
> version number in the header file names, and probably should not
> include it in shared library names when linking or installing.
Except that you have to bind in the soname to the dynamic libraries
exactly so that dynamic linking can ensure version compatibility.
>
> * Different variants may be built and published together, or may be
> built and published at different times. For example, I might build and
> publish the linux variants in one gradle invocation, and then later
> build and publish the windows variants.
Published where? If the intention is to start a repository then this
all might work. Without a repository, it is not clear that dependency
resolution is a tractable problem.
The whole of Autoconf (and hence equivalent features in SCons and Waf)
is about determining whether currently installed dependencies are
sufficient to the task of compiling
> So, that's the problem we want to solve. Of course, I'm sure we can
> chop this into smaller pieces and solve the problem incrementally.
>
>
> Any thoughts you have on either the description of the problem, or
> possible solutions are most welcome. I'll post my thoughts about how
> we solve the problem in some later emails.
Hopefully the above help and are not too obviously already known.
--
Russel.
=============================================================================
Dr Russel Winder t: +44 20 7585 2200 voip: sip:[email protected]
41 Buckmaster Road m: +44 7770 465 077 xmpp: [email protected]
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
signature.asc
Description: This is a digitally signed message part
