Well, I tried upstreaming the new build scripts to some projects and it didn’t go well. Some of the reasons I’ve heard of:
> I installed CMake 2.8.6 five years ago and I don’t want to update yet > again! People relying on old versions is quite common and any attempt > to raise the min version will be frowned upon (see the discussion in > the LLVM mailing lists for example). Spack is really good at installing dependencies, and makes this a LOT easier. In your Spack recipe, you just tell it which version of CMake your package needs. If Spack hasn't already built that version, it will download and install it for you. Building packages by hand, and configuring their dependencies, needs to go the way of stone spears. > We prefer to use autotools and don’t want to have to learn CMake. > That’s fair. But also, no one likes to build an autotools backed > project for Android or iOS. I suppose it's fair. But a Google search of "convert CMake to Autotools" results in 9:1 stories of people abandoning Autotools for CMake. Except for the fact that it works well for users, I can't say enough evil things about Autotools. Part of the benefit of Autotools is it "just works" without requiring the user to install anything. This benefit is of little value once you move to an auto-builder like Spack. The days when you can get any interesting software to work without installing a zillion dependencies first are long gone. > I’ve never heard of Spack before. It looks better than other solutions > I’ve seen before. The great and unique thing about Spack is it can install a zillion versions of each package. For example... if Package B uses MPI, I can build B-1.7 two (or more) times --- once with OpenMPI and once with MPICH. And I can install them side-by-side. If you change any of the dependencies of a package, Spack will see that as a new and separate version. Most auto-builders let you build one software distro, with only one build of each package (or sometimes one build per numerical version of the package or compiler or something). Spack's versioning is a lot more powerful. > But you still have to manage all the options from your build script Not sure what you mean by this. True, there is some redundancy building code. First you put the options and dependencies in a package's CMake build. And then you put them into the Spack build again. Some things could be simplified if we assumed our CMake-based packages would only ever be built with Spack. But we still need to create CMake-based software that can be installed by hand. Hence the redundancy between the CMake build scripts and the Spack package. In practice, this has not been the end of the world. Another nice thing about Spack is there is no difference between your libraries and Third-party libraries. > and publish the binaries somewhere. In its original incarnation, Spack builds from source. It does not publish or install from binary distros (because the build you asked for, with all its dependency variants, is likely not a build that's ever been built before). There's currently work on a project to use Spack to produce binary RPMs, and maybe other forms of binary distribution. > Then you need to teach your build scripts to get the right version. Your build scripts know nothing about Spack. Spack is an auto-builder that sits ON TOP of your build scripts. > I won’t trade my builds from source for a set of prebuilt binaries anytime soon I think :) Spack builds from source, it is not prebuilt binaries. > > I don't think CMake is the best place to do it, > Can you provide any details? I personally think that CMake is a > natural and the only place where it should be done. The most important reason here is because there are combinatorially many versions of a package you COULD install, depending on what versions of its dependencies you link with, and CMake provides nothing to address that issue. See here for an overview of how Spack addresses the combinatorial versioning issue (which no other auto-builder does, to the best of my knowledge): http://hpcugent.github.io/easybuild/files/SC14_BoF_Spack.pdf Once you've built something, it's nice to be able to re-use it. If I have a top-level CMake project that automatically builds three dependencies, will other projects be able to make use of those dependencies I've built? Or do they become private? If libraries cannot be shared between applications, you will get a LOT of library bloat, especially among the low-level libraries that get repeated numerous times. Admittedly, this may not be such an issue in some environments where people are really only focused on building one thing. If you make a project, you might see it as a "top-level" project. But someone else might want to build something bigger on top of your work. You can never assume that "this package is top-level and no one will ever depend on it." Another obvious problem with using CMake for everything is that not all packages come with CMake builds; most do not, in fact. Even if we CAN re-write all the buils into CMake, that is a lot of extra effort. As Florent has discovered, upstream authors do not always see a CMake build in a favorable light, and these re-worked builds are not always as functional as the original. Moreover... writing a Spack recipe is an order of magnitude easier than writing a CMake build. Usually, it's just a matter of calling `configure` or `cmake` with the right options. Although we can maybe imagine a world in which everyone eventually abandons Autotools for CMake, it is still not realistic to expect that Python, Haskell or Java programs will ever come with CMake builds. This would be OK if each language exited in its own silo. But they don't. Python packages (built with setuptools) routinely depend on C-based packages (built with Autotools or CMake). By being agnostic to the build system, auto-builders (like Spack, Macports, HomeBrew, etc) are able to make packages work together, regardless of the build system chosen for each one. In the sense that CMake is a Turing-complete language, there's no fundamental reason you CAN'T write a meta-builder in CMake. But gosh... the language sure is arcane (but still better than Autotools by a long shot). I like to imagine that if CMake were starting off today, it would be written in Python. > It is difficult indeed but it's the only way to reach the goal. Not sure what goal here. My software stack requires about 70 packages. I'm so glad I didn't have to rewrite 70 CMake builds. As long as an Autotools build works (meaning it puts stuff in prefix/bin, prefix/lib, etc. at the end of the day), and I don't have to look at it, why do I care? > Exceptions are big libraries with a lot of components such as Qt or > Boost, it's unmaintainable approach, inevitable evil :( I guess I rely on a lot of evil libraries. > It's not too difficult to support installing multiple parallel > versions of the development environment for different versions of > the product - just have each one install to a different location The problem here is each dev environment is fully repeated, even if only a few of the top-level packages have changed. I heard about one well-known Silicon Valley company that did this, and is maintaining 50 different versions of their Python environment (complete with all their Python libs, both internal and 3d party). They also have special rules about which code you can depend on, and which is designated "top level," meaning you cannot depend on it. Unfortunately, they did not have enough bandwidth or storage in their data center to distribute 50X their dev environment to each node where it needed to be. Spack can help address this problem by allowing sharing of lower-level libraries between different top-level environments. On Tue, Aug 16, 2016 at 9:31 PM, Florent Castelli < [email protected]> wrote: > On 16/08/2016 23:36, Ruslan Baratov wrote: > >> On 16-Aug-16 16:37, Florent Castelli wrote: >> >>> Well, I tried upstreaming the new build scripts to some projects and it >>> didn’t go well. >>> Some of the reasons I’ve heard of: >>> - Windows developpers don’t use CMake, they have project files on the >>> repository. >>> The CMake files for Windows will never be updated. >>> >> They can coexists, it's easier then maintaining forks. If only C++ code >> changed you got new version "for free". >> >> - I installed CMake 2.8.6 five years ago and I don’t want to update yet >>> again! >>> People relying on old versions is quite common and any attempt to >>> raise the min >>> version will be frowned upon (see the discussion in the LLVM mailing >>> lists for example). >>> >> You can add `if(CMAKE_VERSION VERSION_LESS ...)` condition. It's hard to >> support such hairy configuration but anyway. >> >> - We prefer to use autotools and don’t want to have to learn CMake. >>> That’s fair. But also, no one likes to build an autotools backed >>> project for Android or iOS. >>> >> Just for your info Hunter use build scheme for autotools project: >> https://github.com/ruslo/hunter/blob/b4c370e32798cc3da74c37e >> 4156c3bfc77add379/cmake/modules/hunter_autotools_project.cmake >> It can create universal iOS libraries and works for Android. There are a >> lot of efforts made by Alexandre Pretyman so this can be possible, it has >> some peculiarities. I guess he can clarify anything if you need details. >> >> Ruslo >> > > You're addressing the wrong person here. > > /Florent > >
-- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake
