Re: [CMake] Building third party libraries along with normal targets

[Tamás Kenéz]

[about shell script portability]

We do use shell-scripts on Windows since git-bash comes with git for free.
Our workflow is centered around the command-line (git-bash for Windows) and
you can run shell scripts in git-bash, that's why compatibility is not a
problem. All our scripts and workflows are the same on all our platforms
(windows, linux, mac).

This is an issue only if you're not using git on Windows so adding git-bash
to the workflow would be an extra hassle.


On Thu, Mar 30, 2017 at 3:45 PM, Robert Dailey 
wrote:

> On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz 
> wrote:
> > An alternative to the CMake superbuild: leave your actual project intact.
> > Simply create a separate shell script which builds all the dependencies
> (by
> > running cmake commands). Or, if you do like the ExternalProject stuff,
> > create a superbuild which builds only the dependencies.
>
> Problem with that is that it's not a portable solution. The #1 benefit
> of using CMake superbuilds (supplied with CMake build solution for the
> target library) is that it will function correctly on all platforms.
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Elizabeth A. Fischer]

Miller,

Thank you for your input; I'm hoping we can use it to improve our
description of Spack at http://spack.io  .  I'm cross-posting to the Spack
list, maybe someone there can add to this.

https://github.com/LLNL/spack/issues/2115

There has been talk of a comparison; but I'm not familiar enough with the
other systems to talk about them.  I can/will say a bit about Spack.


>
but none actually have defined the problem or their solution.


Spack is aimed at people involved in scientific computing.  This is more a
social distinction than technical: there is nothing particularly special
about scientific software environments, and I know of corporate
environments that are considering Spack as well.  However, this intended
audience does drive the design in some ways.

1. As with Macports, Homebrew, Gentoo, etc. Spack is focused on building
things from source.  There has been some work on packing up Spack builds as
RPMs so they can be quickly installed across multiple machines.

2. Spack supports cross-compilers.  Cross compilation is an intrinsic part
of some supercomputers, in which the login nodes are a different
architecture from the nodes the software runs on.  Maybe a Spacker who has
done cross-compiling can speak up here...

3. Spack supports multiple versions of installed packages --- where the
"version" is actually a hash of the package version, plus the version of
all its dependencies.  Central to Spack is the "concretization" algorithm,
which chooses which version of each package to build for you, based on the
constraints you give it.  For example, you could build 24 versions of
NetCDF, based on different compilers and MPI options you choose.  This kind
of flexibility is incredibly useful in a scientific environment, where
different top-level applications require different versions of various
underlying dependencies.
AFAIK, this concretization algorithm unique among auto-builders;
and it deserves some time to think about (look at the Powerpoint
presentations on the Spack page).  More common are builders (eg, those used
for Linux distros) that produce a *single* version of each desired package
--- and you have to manually specify the version of each package you want.
This is the step that Spack does automatically for you.



> what packages they create,


Spack now has over 1000 packages it knows how to build:

https://github.com/LLNL/spack/tree/develop/var/spack/repos/builtin/packages

NOTE: Be careful when comparing how many packages an auto-builder can
create.  Some package managers, for example, have a different "package" for
each version that might be installed.  For example, *netcdf-1.0*,
*netcdf-1.1*, *netcdf-1.2*, etc.  In Spack, each "package" corresponds to
an upstream piece of software (i.e. just *netcdf)*, and Spack knows how to
install potentially many versions and variants of it*.*


> what compromises they make,



> what they expect of my environment...


Just Python 2.6 or 2.7: do a "git clone", then run, starting up is very
easy.  Spack needs other basic tools as well (*git*, *curl*, etc.).  On
newer systems, they "just work"; on older systems, Spack can be used to
install them.


limitations


1. Spack currently runs on Linux + MacOS, not Windows.

2. Spack doesn't really know how to work with sub-package managers: eg,
using Spack to install `pip`, and then using `pip` to install things.  Same
thing for `luarocks`, `conda`, `node-js`, etc.  Spack *can* install these
things, just don't expect great integration.  We've thought a lot about how
to integrate well, and haven't come up with any home run ideas.  The issue
is most significant with *node-js* when a Spack package wants to depend on
a bunch of Javascript packages.

3. Spack's concretization algorithm, while still much faster than a human,
is too slow to conveniently use for software stacks of greater than ~100
packages.  This also makes it less than ideal for the Javascript world,
where packages are small and numerous.


> the other 10% of the time our system has an imcompatible version of some
> library and things blow up when you try to use some uncommon feature).


Spack goes to great lengths to build all dependencies itself, and not
"accidentally" depend on system-supplied software.

This is getting off-topic for the cmake mailing list, but I don't know
> where else to move the conversation.
>

Feel free to join the Spack mailing list (CC'd above).

-- Elizabeth
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to 

Re: [CMake] Building third party libraries along with normal targets

[Konstantin Tokarev]

30.03.2017, 18:15, "Miller Henry" <millerhe...@johndeere.com>:
> I have found the following projects which all seem to do some variation of a 
> meta build so that you can build multiple projects that depend on each other 
> and manage dependencies. (there are a couple others that seem to not be 
> maintained as well)
>
> https://gradle.org/
> https://bazel.build/
> https://github.com/LLNL/spack
> https://github.com/ruslo/hunter
> http://www.biicode.com
> https://conan.io/
> https://conda.io/

Also, https://github.com/GNOME/jhbuild (notably used by GNOME, WebKit, and 
X.org communities)

>
> Unfortunately I have never found anyone who has actually compared even two of 
> these. None of the projects have good marketing: it appears they somehow 
> solve similar problems, but none actually have defined the problem or their 
> solution. It is like everyone assumes that everyone in the world has their 
> exact same problem and the solution is obvious so the only thing left is the 
> details of implementing it. This of course tells me nothing about if they 
> handle cross compiling (not a common use case but it is yours and mine), what 
> packages they create, what compromises they make, what they expect of my 
> environment... These are important questions: I'm pretty sure that I could 
> eliminate several just by comparing my needs to their features.
>
> I'm currently using an in house system that builds everything in a Docker 
> which lets me ensure nobody is accidentally using the wrong compiler. (we 
> cross compile for a x86 target - 90% of the time if you build with gcc for 
> the local system everything will work just fine, the other 10% of the time 
> our system has an imcompatible version of some library and things blow up 
> when you try to use some uncommon feature). I'm thinking about moving to one 
> of the above, but I haven't actually evaluated anything.
>
> If you do evaluate any of the above please document your experience and in 
> particular what is good/bad about the things you look at.
>
> This is getting off-topic for the cmake mailing list, but I don't know where 
> else to move the conversation.
>
> -Original Message-
> From: CMake [mailto:cmake-boun...@cmake.org] On Behalf Of Robert Dailey
> Sent: Wednesday, March 29, 2017 9:39 PM
> To: Florent Castelli <florent.caste...@gmail.com>
> Cc: CMake <cmake@cmake.org>
> Subject: Re: [CMake] Building third party libraries along with normal targets
>
> On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli 
> <florent.caste...@gmail.com> wrote:
>>  On 30/03/2017 03:54, Robert Dailey wrote:
>>>  On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
>>>  <florent.caste...@gmail.com> wrote:
>>>>  This is known as "super build".
>>>>  Yes, this is exactly why I made my Boost CMake build scripts, which
>>>>  you use unless you changed your mind today :)
>>>
>>>  You mean this?
>>>  https://github.com/Orphis/boost-cmake
>>>
>>>  It's on the drawing board, for sure. It's the best solution I've
>>>  found. It's also part of the reason I'm asking this question to begin
>>>  with.
>>>
>>>  Although, this specific method of building boost doesn't use
>>>  superbuild, it uses normal add_subdirectory() with target.
>>
>>  It doesn't "super build" its dependencies, you have to provide them
>>  externally, but it is a possible addition.
>>  But it is designed to be part of a super build environment and is used
>>  as such at some large companies shipping to millions of people.
>>
>>>>  It can be done for other projects as well. Sometimes, they even
>>>>  provide CMake build scripts you can use directly with
>>>>  "add_subdirectory()" so you don't have to write CMake scripts or use
>>>>  "ExternalProject_Add()" (which isn't all great since it doesn't
>>>>  propagate all your current project settings).
>>>
>>>  Long term, I'm thinking a project like hunter[1] would be the best
>>>  choice. The problem with hunter is that it doesn't separate concerns
>>>  between host machine and target host. Specifically, when cross
>>>  compiling.
>>>
>>>  All the superbuild commands assume linux command syntax, simply
>>>  because I specified a toolchain utilizing android NDK. For this
>>>  reason it requires a lot of TLC to get working.
>>>
>>>  [1]: https://github.com/ruslo/hunter
>>
>>  There are those issues and also ABI changing flags that can become an
>>  issue as they aren't always propagated f

Re: [CMake] Building third party libraries along with normal targets

[Miller Henry]

I have found the following projects which all seem to do some variation of a 
meta build so that you can build multiple projects that depend on each other 
and manage dependencies.  (there are a couple others that seem to not be 
maintained as well)

https://gradle.org/
https://bazel.build/ 
https://github.com/LLNL/spack 
https://github.com/ruslo/hunter 
http://www.biicode.com 
https://conan.io/
https://conda.io/

Unfortunately I have never found anyone who has actually compared even two of 
these. None of the projects have good marketing: it appears they somehow solve 
similar problems, but none actually have defined the problem or their solution. 
 It is like everyone assumes that everyone in the world has their exact same 
problem and the solution is obvious so the only thing left is the details of 
implementing it.  This of course tells me nothing about if they handle cross 
compiling (not a common use case but it is yours and mine), what packages they 
create, what compromises they make, what they expect of my environment...  
These are important questions: I'm pretty sure that I could eliminate several 
just by comparing my needs to their features.

I'm currently using an in house system that builds everything in a Docker which 
lets me ensure nobody is accidentally using the wrong compiler. (we cross 
compile for a x86 target - 90% of the time if you build with gcc for the local 
system everything will work just fine, the other 10% of the time our system has 
an imcompatible version of some library and things blow up when you try to use 
some uncommon feature). I'm thinking about moving to one of the above, but I 
haven't actually evaluated anything.

If you do evaluate any of the above please document your experience and in 
particular what is good/bad about the things you look at.

This is getting off-topic for the cmake mailing list, but I don't know where 
else to move the conversation.


-Original Message-
From: CMake [mailto:cmake-boun...@cmake.org] On Behalf Of Robert Dailey
Sent: Wednesday, March 29, 2017 9:39 PM
To: Florent Castelli <florent.caste...@gmail.com>
Cc: CMake <cmake@cmake.org>
Subject: Re: [CMake] Building third party libraries along with normal targets

On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli <florent.caste...@gmail.com> 
wrote:
> On 30/03/2017 03:54, Robert Dailey wrote:
>>
>> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli 
>> <florent.caste...@gmail.com> wrote:
>>>
>>> This is known as "super build".
>>> Yes, this is exactly why I made my Boost CMake build scripts, which 
>>> you use unless you changed your mind today :)
>>
>> You mean this?
>> https://github.com/Orphis/boost-cmake
>>
>> It's on the drawing board, for sure. It's the best solution I've 
>> found. It's also part of the reason I'm asking this question to begin 
>> with.
>>
>> Although, this specific method of building boost doesn't use 
>> superbuild, it uses normal add_subdirectory() with target.
>
> It doesn't "super build" its dependencies, you have to provide them 
> externally, but it is a possible addition.
> But it is designed to be part of a super build environment and is used 
> as such at some large companies shipping to millions of people.
>
>>
>>> It can be done for other projects as well. Sometimes, they even 
>>> provide CMake build scripts you can use directly with 
>>> "add_subdirectory()" so you don't have to write CMake scripts or use 
>>> "ExternalProject_Add()" (which isn't all great since it doesn't 
>>> propagate all your current project settings).
>>
>> Long term, I'm thinking a project like hunter[1] would be the best 
>> choice. The problem with hunter is that it doesn't separate concerns 
>> between host machine and target host. Specifically, when cross 
>> compiling.
>>
>> All the superbuild commands assume linux command syntax, simply 
>> because I specified a toolchain utilizing android NDK. For this 
>> reason it requires a lot of TLC to get working.
>>
>> [1]: https://github.com/ruslo/hunter
>
> There are those issues and also ABI changing flags that can become an 
> issue as they aren't always propagated from what I can see.
>
>>> You can use ccache or its Windows variants to make it faster. But 
>>> you also shouldn't need "ninja clean" most of the time. Possibly, 
>>> you could just clean a specific target "ninja -t clean foo".
>>> If you declare all your dependencies properly, then you could just 
>>> always run "ninja" and the build will just be correct.
>>> If your purpose is to see the compiler output again (to fix 
>>> warnings), it is ac

Re: [CMake] Building third party libraries along with normal targets

[Elizabeth A. Fischer]

I do this using the spack autobuilder.  Only problem is it doesn't run on
windows.  Maybe Conda?
On Mar 30, 2017 9:45 AM, "Robert Dailey"  wrote:

> On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz 
> wrote:
> > An alternative to the CMake superbuild: leave your actual project intact.
> > Simply create a separate shell script which builds all the dependencies
> (by
> > running cmake commands). Or, if you do like the ExternalProject stuff,
> > create a superbuild which builds only the dependencies.
>
> Problem with that is that it's not a portable solution. The #1 benefit
> of using CMake superbuilds (supplied with CMake build solution for the
> target library) is that it will function correctly on all platforms.
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Robert Dailey]

On Thu, Mar 30, 2017 at 3:42 AM, Tamás Kenéz  wrote:
> An alternative to the CMake superbuild: leave your actual project intact.
> Simply create a separate shell script which builds all the dependencies (by
> running cmake commands). Or, if you do like the ExternalProject stuff,
> create a superbuild which builds only the dependencies.

Problem with that is that it's not a portable solution. The #1 benefit
of using CMake superbuilds (supplied with CMake build solution for the
target library) is that it will function correctly on all platforms.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Tamás Kenéz]

An alternative to the CMake superbuild: leave your actual project intact.
Simply create a separate shell script which builds all the dependencies (by
running cmake commands). Or, if you do like the ExternalProject stuff,
create a superbuild which builds only the dependencies.

Tamas

On Thu, Mar 30, 2017 at 5:01 AM, Florent Castelli <
florent.caste...@gmail.com> wrote:

> On 30/03/2017 04:38, Robert Dailey wrote:
>
>> At the end of the day, I've solved the Boost problem but I have many
>> other libraries I still manually build: openssl, libpng, zlib, etc.
>> It's as you said, maintaining build scripts for all of these will be
>> challenging but I think that's the proper way to go in my specific
>> situation (multi platform). It'll be less maintenance than rebuilding
>> binaries 3x per each.
>>
>> Do you know how to set up the boost cmake project I linked on github
>> to work with superbuild? And do you prefer your superbuild execution
>> to happen during cmake generate (basically cmake invokes itself for
>> the superbuild) or do you prefer a separate external run, independent
>> of the normal one (basically 2x cmake invocations)?
>>
>> Do you know of similar projects for other libraries I mentioned? Could
>> hand write my own, but seems better to piggyback on someone else's
>> efforts.
>>
>> Thanks for your feedback!
>>
>
> Only a few libraries are tricky to build, libpng or zlib are trivial for
> example.
>
> A super-build is just having one big project and defining all the targets
> directly in there, using external CMake build scripts through
> add_subdirectory() for example. There is just one CMake invocation.
> This is how my Boost CMake project (yes, I'm the one being it) works. As
> for the Boost dependencies, are you really using them? If you're not, then
> you don't need to build the parts of Boost using them, most are optional.
> If you do, then you can provide Find*.cmake scripts returning targets
> you've built yourself currently. Feel free to create issues on the bug
> tracker if you need assistance with that.
>
> I don't know on the top of my head similar projects for all libraries, but
> I know there are many people writing CMake build scripts for all kind of
> libraries, so just search on the web :)
>
> /Florent
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensou
> rce/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Florent Castelli]

On 30/03/2017 04:38, Robert Dailey wrote:

At the end of the day, I've solved the Boost problem but I have many
other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be
challenging but I think that's the proper way to go in my specific
situation (multi platform). It'll be less maintenance than rebuilding
binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github
to work with superbuild? And do you prefer your superbuild execution
to happen during cmake generate (basically cmake invokes itself for
the superbuild) or do you prefer a separate external run, independent
of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could
hand write my own, but seems better to piggyback on someone else's
efforts.

Thanks for your feedback!


Only a few libraries are tricky to build, libpng or zlib are trivial for 
example.


A super-build is just having one big project and defining all the 
targets directly in there, using external CMake build scripts through 
add_subdirectory() for example. There is just one CMake invocation.
This is how my Boost CMake project (yes, I'm the one being it) works. As 
for the Boost dependencies, are you really using them? If you're not, 
then you don't need to build the parts of Boost using them, most are 
optional.
If you do, then you can provide Find*.cmake scripts returning targets 
you've built yourself currently. Feel free to create issues on the bug 
tracker if you need assistance with that.


I don't know on the top of my head similar projects for all libraries, 
but I know there are many people writing CMake build scripts for all 
kind of libraries, so just search on the web :)


/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Robert Dailey]

On Wed, Mar 29, 2017 at 9:32 PM, Florent Castelli
 wrote:
> On 30/03/2017 03:54, Robert Dailey wrote:
>>
>> On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
>>  wrote:
>>>
>>> This is known as "super build".
>>> Yes, this is exactly why I made my Boost CMake build scripts, which you
>>> use
>>> unless you changed your mind today :)
>>
>> You mean this?
>> https://github.com/Orphis/boost-cmake
>>
>> It's on the drawing board, for sure. It's the best solution I've
>> found. It's also part of the reason I'm asking this question to begin
>> with.
>>
>> Although, this specific method of building boost doesn't use
>> superbuild, it uses normal add_subdirectory() with target.
>
> It doesn't "super build" its dependencies, you have to provide them
> externally, but it is a possible addition.
> But it is designed to be part of a super build environment and is used as
> such at some large companies shipping to millions of people.
>
>>
>>> It can be done for other projects as well. Sometimes, they even provide
>>> CMake build scripts you can use directly with "add_subdirectory()" so you
>>> don't have to write CMake scripts or use "ExternalProject_Add()" (which
>>> isn't all great since it doesn't propagate all your current project
>>> settings).
>>
>> Long term, I'm thinking a project like hunter[1] would be the best
>> choice. The problem with hunter is that it doesn't separate concerns
>> between host machine and target host. Specifically, when cross
>> compiling.
>>
>> All the superbuild commands assume linux command syntax, simply
>> because I specified a toolchain utilizing android NDK. For this reason
>> it requires a lot of TLC to get working.
>>
>> [1]: https://github.com/ruslo/hunter
>
> There are those issues and also ABI changing flags that can become an issue
> as they aren't always propagated from what I can see.
>
>>> You can use ccache or its Windows variants to make it faster. But you
>>> also
>>> shouldn't need "ninja clean" most of the time. Possibly, you could just
>>> clean a specific target "ninja -t clean foo".
>>> If you declare all your dependencies properly, then you could just always
>>> run "ninja" and the build will just be correct.
>>> If your purpose is to see the compiler output again (to fix warnings), it
>>> is
>>> acceptable to clean and rebuild (with ccache it should be fast enough).
>>> Personally, I just have a very long history in my terminal and scroll
>>> back
>>> or pipe the build content to a file to look at it later. Some IDEs will
>>> also
>>> record all the compilation output and make it available later (Xcode does
>>> it), then it's less of an issue.
>>
>> Never used ccache before, what does it do? Also if even if you ninja
>> -t to clean, it will also clean dependencies. So if I specify those
>> in-project dependencies that actually build third party libs, those
>> will be cleaned too.
>
>
> It's a program you use to record compilation outputs and save them to a disk
> cache. If you build a file again, instead of compiling it, it will just
> fetch the outputs from the cache and return the same warnings. It's usually
> a safe optimization in a build pipeline and is quite easy to use with CMake
> and Ninja generator.
>
> It works well with GCC or Clang and there are some alternative versions for
> the Microsoft compiler too.

At the end of the day, I've solved the Boost problem but I have many
other libraries I still manually build: openssl, libpng, zlib, etc.
It's as you said, maintaining build scripts for all of these will be
challenging but I think that's the proper way to go in my specific
situation (multi platform). It'll be less maintenance than rebuilding
binaries 3x per each.

Do you know how to set up the boost cmake project I linked on github
to work with superbuild? And do you prefer your superbuild execution
to happen during cmake generate (basically cmake invokes itself for
the superbuild) or do you prefer a separate external run, independent
of the normal one (basically 2x cmake invocations)?

Do you know of similar projects for other libraries I mentioned? Could
hand write my own, but seems better to piggyback on someone else's
efforts.

Thanks for your feedback!
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Florent Castelli]

On 30/03/2017 03:54, Robert Dailey wrote:

On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
 wrote:

This is known as "super build".
Yes, this is exactly why I made my Boost CMake build scripts, which you use
unless you changed your mind today :)

You mean this?
https://github.com/Orphis/boost-cmake

It's on the drawing board, for sure. It's the best solution I've
found. It's also part of the reason I'm asking this question to begin
with.

Although, this specific method of building boost doesn't use
superbuild, it uses normal add_subdirectory() with target.
It doesn't "super build" its dependencies, you have to provide them 
externally, but it is a possible addition.
But it is designed to be part of a super build environment and is used 
as such at some large companies shipping to millions of people.





It can be done for other projects as well. Sometimes, they even provide
CMake build scripts you can use directly with "add_subdirectory()" so you
don't have to write CMake scripts or use "ExternalProject_Add()" (which
isn't all great since it doesn't propagate all your current project
settings).

Long term, I'm thinking a project like hunter[1] would be the best
choice. The problem with hunter is that it doesn't separate concerns
between host machine and target host. Specifically, when cross
compiling.

All the superbuild commands assume linux command syntax, simply
because I specified a toolchain utilizing android NDK. For this reason
it requires a lot of TLC to get working.

[1]: https://github.com/ruslo/hunter
There are those issues and also ABI changing flags that can become an 
issue as they aren't always propagated from what I can see.



You can use ccache or its Windows variants to make it faster. But you also
shouldn't need "ninja clean" most of the time. Possibly, you could just
clean a specific target "ninja -t clean foo".
If you declare all your dependencies properly, then you could just always
run "ninja" and the build will just be correct.
If your purpose is to see the compiler output again (to fix warnings), it is
acceptable to clean and rebuild (with ccache it should be fast enough).
Personally, I just have a very long history in my terminal and scroll back
or pipe the build content to a file to look at it later. Some IDEs will also
record all the compilation output and make it available later (Xcode does
it), then it's less of an issue.

Never used ccache before, what does it do? Also if even if you ninja
-t to clean, it will also clean dependencies. So if I specify those
in-project dependencies that actually build third party libs, those
will be cleaned too.


It's a program you use to record compilation outputs and save them to a 
disk cache. If you build a file again, instead of compiling it, it will 
just fetch the outputs from the cache and return the same warnings. It's 
usually a safe optimization in a build pipeline and is quite easy to use 
with CMake and Ninja generator.


It works well with GCC or Clang and there are some alternative versions 
for the Microsoft compiler too.


/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Robert Dailey]

On Wed, Mar 29, 2017 at 8:18 PM, Florent Castelli
 wrote:
> This is known as "super build".
> Yes, this is exactly why I made my Boost CMake build scripts, which you use
> unless you changed your mind today :)

You mean this?
https://github.com/Orphis/boost-cmake

It's on the drawing board, for sure. It's the best solution I've
found. It's also part of the reason I'm asking this question to begin
with.

Although, this specific method of building boost doesn't use
superbuild, it uses normal add_subdirectory() with target.

> It can be done for other projects as well. Sometimes, they even provide
> CMake build scripts you can use directly with "add_subdirectory()" so you
> don't have to write CMake scripts or use "ExternalProject_Add()" (which
> isn't all great since it doesn't propagate all your current project
> settings).

Long term, I'm thinking a project like hunter[1] would be the best
choice. The problem with hunter is that it doesn't separate concerns
between host machine and target host. Specifically, when cross
compiling.

All the superbuild commands assume linux command syntax, simply
because I specified a toolchain utilizing android NDK. For this reason
it requires a lot of TLC to get working.

[1]: https://github.com/ruslo/hunter

> You can use ccache or its Windows variants to make it faster. But you also
> shouldn't need "ninja clean" most of the time. Possibly, you could just
> clean a specific target "ninja -t clean foo".
> If you declare all your dependencies properly, then you could just always
> run "ninja" and the build will just be correct.
> If your purpose is to see the compiler output again (to fix warnings), it is
> acceptable to clean and rebuild (with ccache it should be fast enough).
> Personally, I just have a very long history in my terminal and scroll back
> or pipe the build content to a file to look at it later. Some IDEs will also
> record all the compilation output and make it available later (Xcode does
> it), then it's less of an issue.

Never used ccache before, what does it do? Also if even if you ninja
-t to clean, it will also clean dependencies. So if I specify those
in-project dependencies that actually build third party libs, those
will be cleaned too.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Building third party libraries along with normal targets

[Florent Castelli]

On 30/03/2017 02:10, Robert Dailey wrote:

Interested in hearing everyone's thoughts on this idea of mine.

Right now I have several third party libraries: openssl, boost,
libpng, zlib, etc. List goes on. I need to support these libraries on
at least 3 different platforms: ARM android, x86 linux, x86 windows.
It's a real pain in the rear to build each of these libraries a total
of 3 times (once per platform). It makes upgrades painful.

So I was thinking: If I set up each third party library to build from
source as a normal target with the rest of my targets, this would get
me the libraries for each third party lib "for free". Basically, they
would all build the same way no matter what platform I'm on. I can
move to other platforms or architectures in the future with no extra
effort.

This is known as "super build".
Yes, this is exactly why I made my Boost CMake build scripts, which you 
use unless you changed your mind today :)
It can be done for other projects as well. Sometimes, they even provide 
CMake build scripts you can use directly with "add_subdirectory()" so 
you don't have to write CMake scripts or use "ExternalProject_Add()" 
(which isn't all great since it doesn't propagate all your current 
project settings).



The downside, of course, is that they build along with my normal
targets. Which means doing "ninja clean" will clean them, when I
really don't need to, and it will make build times get longer.
You can use ccache or its Windows variants to make it faster. But you 
also shouldn't need "ninja clean" most of the time. Possibly, you could 
just clean a specific target "ninja -t clean foo".
If you declare all your dependencies properly, then you could just 
always run "ninja" and the build will just be correct.
If your purpose is to see the compiler output again (to fix warnings), 
it is acceptable to clean and rebuild (with ccache it should be fast 
enough).
Personally, I just have a very long history in my terminal and scroll 
back or pipe the build content to a file to look at it later. Some IDEs 
will also record all the compilation output and make it available later 
(Xcode does it), then it's less of an issue.



Any thoughts on this idea? Are there any other downsides? Is this the
best way to have library support across platforms?


The downside is that you have to maintain lots of 3rd party build 
scripts. Depending on the side of the project, it might be a big 
overhead for the team. But it might also be totally worth it in order to 
support a lot of platforms, it just makes it easier than handling a huge 
matrix of binaries for each platform, arch and compiler and compiler 
options.


"Hey, you want to try LTO? Just add the flag and it's done!".

/Florent

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake