Re: [CMake] Need ideas/opinions on third party library management

[Daniel Schepler]

For what it's worth, what I've done here is to create binary packages of each 
third-party library for each supported platform (i.e. .deb packages for Ubuntu, 
.rpm packages for RHEL/CentOS, Chocolatey packages for Windows).  Except for 
cases where the system already provided sufficient versions of some of the 
libraries.  At the bottom of the internal dependency chain is a base package 
which sets up the infrastructure (e.g. an /etc/profile.d file to add a shell 
function to add entries to CMAKE_PREFIX_PATH, LD_LIBRARY_PATH, etc.) and at the 
top is a set of dependency metapackages, one for each product we have to build. 
 And yes, the middle layer has a combination of large packages like Qt and 
Boost with smaller packages like Expat and SQLite.

Pluses:
- Extremely convenient for an internal developer to just follow a few 
instructions to add our internal repository, then run "apt-get install 
anchorage-exata-deps".
- Similarly convenient for customers developing custom code on top of our code 
if we provide an archive of the required packages, then they can point 
apt/yum/chocolatey to a file: URL.
- It's not too difficult to support installing multiple parallel versions of 
the development environment for different versions of the product - just have 
each one install to a different location, and then the developer selects what 
to use by running the appropriate shell function.
- Greatly simplifies the CMake files for our code, since you just use the 
standard find_package() command (possibly requiring us to write a Find*.cmake 
module).
- Flexibility of being able to easily experiment with a new or custom-compiled 
version of one of the dependencies.
- If an internal developer happens to prefer working on a platform which we 
don't officially support (e.g. Debian testing or Mint), it's generally easy to 
git clone my packaging repositories and build packages for yourself.

Minuses:
- Duplicate packaging work for each different packaging tool when adding a new 
third-party library.
- I haven't yet automated the process of building binary packages and uploading 
to our repository; so when updating any package to a new version (or adding a 
local patch) there's a lot of manual work to do.  (dupload and reprepro help a 
bit for the Ubuntu builds, and nuget has a built-in facility to upload to our 
server; but I haven't found anything similar for the RPM builds.)
- When we do a release and then need to "fork" the repository to a new 
location, it requires a rebuild of all packages at least on Linux.

Overall, most of the pluses apply to everybody, whereas the minuses are just a 
burden on me (or a future maintainer of the repository).  So overall, it's been 
a big improvement over what we used to do, which was to try to maintain binary 
builds of the third-party libraries in SVN.
--
Daniel Schepler
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Ruslan Baratov via CMake]

On 13-Aug-16 03:12, Elizabeth A. Fischer wrote:
I don't think CMake is the best place to do it, for a number of 
reasons.  I would not try to re-invent the wheel here.


Can you provide any details? I personally think that CMake is a natural 
and the only place where it should be done.



On 16-Aug-16 15:52, Elizabeth A. Fischer wrote:
But any approach that requires every build to be ported to CMake will 
be difficult and labor-prone to scale.  Writing a meta-build recipe is 
usually much easier.


It is difficult indeed but it's the only way to reach the goal. I have 
kind of a meta-build recipe for the Boost, it was the first non-cmake 
recipe I've added and it's almost 3 year of improvements behind but it's 
still far from what I get in CMake from the box (of course it's not 3 
years of development of just this recipe, but anyway). Just take a look 
at this code:


  add_executable(foo foo.cpp)
  add_library(boo boo.cpp)

With CMake:

   No options: you got executable and static library on host
   BUILD_SHARED_LIBS: you got shared library
   CMAKE_MACOSX_BUNDLE: you got bundle instead of executable
   CMAKE_TOOLCHAIN_FILE=android.cmake: you got library for Android
   CMAKE_TOOLCHAIN_FILE=ios.cmake: you got armv7 library for iOS
   CMAKE_TOOLCHAIN_FILE=ios.cmake + CMAKE_IOS_INSTALL_COMBINED: you got
   armv7 + arm64 + i386 + x86_64 universal library
   /* your toolchain file here */: ...
   Debug: you compile with -O0 for *nix or /Od for MSVC
   MinSizeRel: you compile with -Os for *nix or /O1
   /* your custom build type here */: ...
   Any usage requirements? Propagated AUTOMATICALLY to any project that
   use your library: compiler flags, definitions, dependent libraries, etc.
   Optional dependencies? No problems, usage requirements propagated
   too. Same code! No changes needed.
   Usage requirements change from one version to another? Again, no
   problems.

So I choose to add CMakeLists.txt almost in all cases when new package 
needed. Exceptions are big libraries with a lot of components such as Qt 
or Boost, it's unmaintainable approach, inevitable evil :(


Ruslo

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Robert Dailey]

On Fri, Aug 12, 2016 at 4:58 PM, Roger Leigh  wrote:
> You can simplify this to two steps:
>
> 1. Clone the superbuild repository
> 2. Build the third-party and first-party packages
>
> This is the approach we take with the OME super-build
> https://github.com/ome/ome-cmake-superbuild/
>
> See the packages in
> https://github.com/ome/ome-cmake-superbuild/tree/develop/packages
> all the ones with an "ome-" prefix are the first-party libraries;
> everything else is third-party.  They are all external projects.
>
> We have cmake options to select all packages, first-party packages only or
> select individual packages by hand; the recursive dependency resolution will
> add/ignore dependencies depending upon the chosen policy (defaulting to
> include third-party dependencies).  So when you're building on BSD or Linux,
> you can say "no third-party dependencies except for gtest", or choose
> everything, and on Windows where there is no package manager, you always
> build everything.  And on Windows you can build just the dependencies and
> reuse them later; it has support for using cached builds which we use on the
> CI side to cut down build times; we only rebuild the cache when the
> superbuild git tree hash changes; see the scripts/jenkins-build script
>
> In addition to that, we have the means to build the first-party libraries in
> different ways
>
> 1. From source release tarball (last stable release)
> 2. From git (specify remote+branch as cmake options)
> 3. From a local git checkout (on whatever branch is currently checked out)
>
> You can see the selection logic here:
> https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L4
>
> In practice (1) is for end-users to build the whole thing, and (3) is what I
> use in practice to develop locally where I have a full set of git repos
> checked out locally, and I tell the superbuild to build them, along with all
> the third-party deps; (3) is also what we use on the CI system, with jenkins
> cloning all the individual repos into the workspace, then doing pretty much
> the same thing.  (2) is OK for one-off builds, but the need for a full git
> clone for every package is too much for routine use
>
> And you can see how first- and third-party dependencies are handled here:
> https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L48
>
> I might rearrange that so that the classification is in the dependency
> rather than the dependeee later on though.  It kind of evolved this way as
> we added additional features and needs a little cleanup to make it perfect.
>
> Everything in the above repository is BSD licensed, so feel free to use it
> for inspiration or take what you like.  The main point I wanted to make was
> that with the superbuild *everything* can be an external project, including
> your own code.  And from that point you can add in features like being able
> to selectively build just the third-party or just your own code, rather than
> forcing them to be separate manual steps.

I get where you are going with this, but I don't really think our code
base would benefit from treating third party and first party libraries
the same in CMake. We can't really use our different libraries in
isolation anyway, mostly due to a combination of architectural issues
and business logic.

Also I have to do some custom target stuff to pull everything together
for our Android APK build (our product is targeted for Android
platform). I have to do a lot of trickery to integrate with ant to
make our builds seamless between java & C++.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Robert Dailey]

Wow I actually completely forgot about that lol. I think I was looking
into it for some other reasons, not related to work. I will have to
look into it again. I don't really remember much about it.

Thanks for the reminder.

On Sat, Aug 13, 2016 at 7:02 PM, Ruslan Baratov
 wrote:
> Hi, Robert
>
> According to your GitHub account you've send a trivial patch about a year
> ago to the Hunter (https://github.com/ruslo/hunter) package manager. So I
> wonder what is your experience, have you tried it? Have you run into some
> troubles?
>
> Thanks, Ruslo
>
>
> On 12-Aug-16 22:59, Robert Dailey wrote:
>>
>> Hello,
>>
>> There is an internal C++ product at the company I work for which I
>> have written a series of CMake scripts for. This project actually has
>> dependencies on several open source libraries, such as boost,
>> freetype, openssl, etc.
>>
>> Right now what we do is build each of these third party libraries *by
>> hand*, once for every platform we support (Windows, Linux x86, Android
>> NDK). Then we stuff the includes (headers) and libraries
>> (static/shared) in a submodule and the primary code base's CMake
>> scripts pull them in as interface targets.
>>
>> This works well and is light-weight but is a pain when upgrading or
>> changing libraries. It's a pain because if I want to upgrade boost, I
>> have to build it up to 6 times (once for each platform and once for
>> each configuration).
>>
>> I've been thinking of a different approach for a while. I've done some
>> toying around with the "Super Build" concept, where I have a separate
>> CMake project that does nothing but use the ExternalProject module to
>> build libraries in real time along with our project. So the order of
>> operations would be as follows (for our automated build server):
>>
>> 1. Clone our "Third Party" repository
>> 2. Use CMake to generate & build the "Super Build" project (this
>> builds boost, openssl, freetype, etc for the current platform).
>> 3. Clone the main code base's repository
>> 4. Use CMake to generate & build, using find_package() to refer to
>> interface targets exported by those third party libraries built in
>> step 2
>>
>> Obviously this will make builds take significantly longer, because
>> we're constantly rebuilding the same third party libraries over and
>> over again. However, it virtually eliminates the maintenance burden
>> for third party libraries because they are built inherently with
>> everything else.
>>
>> Note that I can't refer to pre-built libraries in our build
>> environments because we need very specific control over the versions
>> of our libraries as well as the toolchains that were used to build
>> them. Also we may specifically build our libraries a certain way (such
>> as boost). For this reason we do not rely on our external environment
>> or external package managers to fulfill third party dependencies, like
>> most open source projects do on Linux for example.
>>
>> Does this "Super Build" approach sound like a better idea? What other
>> options are available? The downside with the "Super Build" solution is
>> that it will become very difficult to make the transition between
>> building third party and building our code base seamless. I can't do
>> both in the same generate step because find_package() can't be called
>> until the libraries are built & installed.
>
>
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Ruslan Baratov via CMake]

Hi, Robert

According to your GitHub account you've send a trivial patch about a 
year ago to the Hunter (https://github.com/ruslo/hunter) package 
manager. So I wonder what is your experience, have you tried it? Have 
you run into some troubles?


Thanks, Ruslo

On 12-Aug-16 22:59, Robert Dailey wrote:

Hello,

There is an internal C++ product at the company I work for which I
have written a series of CMake scripts for. This project actually has
dependencies on several open source libraries, such as boost,
freetype, openssl, etc.

Right now what we do is build each of these third party libraries *by
hand*, once for every platform we support (Windows, Linux x86, Android
NDK). Then we stuff the includes (headers) and libraries
(static/shared) in a submodule and the primary code base's CMake
scripts pull them in as interface targets.

This works well and is light-weight but is a pain when upgrading or
changing libraries. It's a pain because if I want to upgrade boost, I
have to build it up to 6 times (once for each platform and once for
each configuration).

I've been thinking of a different approach for a while. I've done some
toying around with the "Super Build" concept, where I have a separate
CMake project that does nothing but use the ExternalProject module to
build libraries in real time along with our project. So the order of
operations would be as follows (for our automated build server):

1. Clone our "Third Party" repository
2. Use CMake to generate & build the "Super Build" project (this
builds boost, openssl, freetype, etc for the current platform).
3. Clone the main code base's repository
4. Use CMake to generate & build, using find_package() to refer to
interface targets exported by those third party libraries built in
step 2

Obviously this will make builds take significantly longer, because
we're constantly rebuilding the same third party libraries over and
over again. However, it virtually eliminates the maintenance burden
for third party libraries because they are built inherently with
everything else.

Note that I can't refer to pre-built libraries in our build
environments because we need very specific control over the versions
of our libraries as well as the toolchains that were used to build
them. Also we may specifically build our libraries a certain way (such
as boost). For this reason we do not rely on our external environment
or external package managers to fulfill third party dependencies, like
most open source projects do on Linux for example.

Does this "Super Build" approach sound like a better idea? What other
options are available? The downside with the "Super Build" solution is
that it will become very difficult to make the transition between
building third party and building our code base seamless. I can't do
both in the same generate step because find_package() can't be called
until the libraries are built & installed.



--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Elizabeth A. Fischer]

I would look into Anaconda, which does work for Windows.  Its version
management is not as well developed as Spack, but it's more cross-platform.

Auto-builders are just coming into their own, it's a brave new world.  I
expect things to be more complete in a few years.

-- Elizabeth


On Sat, Aug 13, 2016 at 7:00 PM, Robert Dailey 
wrote:

> I did some brief digging into spack, and it doesn't look like it
> supports Windows. All I see are shell scripts and the documentation
> uses POSIX.
>
> If I'm going to use a package manager, it needs to be able to support
> Android (ARM), Windows, and Linux. I have specific toolchains that
> I'll need the package manager to use for each of these platforms,
> assuming it compiles these libraries. It also needs to be capable of
> cross-compiling (in the case of ARM toolchain in the Android NDK).
>
> I mean, we can't really call it "reinventing the wheel" if my
> requirements are so specific that no current tooling can support it.
> If you have any info on spack related to my requirements please let me
> know. It looks promising, but so far doesn't seem like it will work
> out.
>
> On Fri, Aug 12, 2016 at 3:41 PM, Elizabeth A. Fischer
>  wrote:
> > This is what Spack and other meta builders do.  See http://github.
> > com/llnl/spack
> >
> > On Aug 12, 2016 3:59 PM, "Robert Dailey" 
> wrote:
> >>
> >> Hello,
> >>
> >> There is an internal C++ product at the company I work for which I
> >> have written a series of CMake scripts for. This project actually has
> >> dependencies on several open source libraries, such as boost,
> >> freetype, openssl, etc.
> >>
> >> Right now what we do is build each of these third party libraries *by
> >> hand*, once for every platform we support (Windows, Linux x86, Android
> >> NDK). Then we stuff the includes (headers) and libraries
> >> (static/shared) in a submodule and the primary code base's CMake
> >> scripts pull them in as interface targets.
> >>
> >> This works well and is light-weight but is a pain when upgrading or
> >> changing libraries. It's a pain because if I want to upgrade boost, I
> >> have to build it up to 6 times (once for each platform and once for
> >> each configuration).
> >>
> >> I've been thinking of a different approach for a while. I've done some
> >> toying around with the "Super Build" concept, where I have a separate
> >> CMake project that does nothing but use the ExternalProject module to
> >> build libraries in real time along with our project. So the order of
> >> operations would be as follows (for our automated build server):
> >>
> >> 1. Clone our "Third Party" repository
> >> 2. Use CMake to generate & build the "Super Build" project (this
> >> builds boost, openssl, freetype, etc for the current platform).
> >> 3. Clone the main code base's repository
> >> 4. Use CMake to generate & build, using find_package() to refer to
> >> interface targets exported by those third party libraries built in
> >> step 2
> >>
> >> Obviously this will make builds take significantly longer, because
> >> we're constantly rebuilding the same third party libraries over and
> >> over again. However, it virtually eliminates the maintenance burden
> >> for third party libraries because they are built inherently with
> >> everything else.
> >>
> >> Note that I can't refer to pre-built libraries in our build
> >> environments because we need very specific control over the versions
> >> of our libraries as well as the toolchains that were used to build
> >> them. Also we may specifically build our libraries a certain way (such
> >> as boost). For this reason we do not rely on our external environment
> >> or external package managers to fulfill third party dependencies, like
> >> most open source projects do on Linux for example.
> >>
> >> Does this "Super Build" approach sound like a better idea? What other
> >> options are available? The downside with the "Super Build" solution is
> >> that it will become very difficult to make the transition between
> >> building third party and building our code base seamless. I can't do
> >> both in the same generate step because find_package() can't be called
> >> until the libraries are built & installed.
> >> --
> >>
> >> Powered by www.kitware.com
> >>
> >> Please keep messages on-topic and check the CMake FAQ at:
> >> http://www.cmake.org/Wiki/CMake_FAQ
> >>
> >> Kitware offers various services to support the CMake community. For more
> >> information on each offering, please visit:
> >>
> >> CMake Support: http://cmake.org/cmake/help/support.html
> >> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> >> CMake Training Courses: http://cmake.org/cmake/help/training.html
> >>
> >> Visit other Kitware open-source projects at
> >> http://www.kitware.com/opensource/opensource.html
> >>
> >> Follow this link to subscribe/unsubscribe:
> >> http://public.kitware.com/mailman/listinfo/cmake
>
-- 

Re: [CMake] Need ideas/opinions on third party library management

[Robert Dailey]

I did some brief digging into spack, and it doesn't look like it
supports Windows. All I see are shell scripts and the documentation
uses POSIX.

If I'm going to use a package manager, it needs to be able to support
Android (ARM), Windows, and Linux. I have specific toolchains that
I'll need the package manager to use for each of these platforms,
assuming it compiles these libraries. It also needs to be capable of
cross-compiling (in the case of ARM toolchain in the Android NDK).

I mean, we can't really call it "reinventing the wheel" if my
requirements are so specific that no current tooling can support it.
If you have any info on spack related to my requirements please let me
know. It looks promising, but so far doesn't seem like it will work
out.

On Fri, Aug 12, 2016 at 3:41 PM, Elizabeth A. Fischer
 wrote:
> This is what Spack and other meta builders do.  See http://github.
> com/llnl/spack
>
> On Aug 12, 2016 3:59 PM, "Robert Dailey"  wrote:
>>
>> Hello,
>>
>> There is an internal C++ product at the company I work for which I
>> have written a series of CMake scripts for. This project actually has
>> dependencies on several open source libraries, such as boost,
>> freetype, openssl, etc.
>>
>> Right now what we do is build each of these third party libraries *by
>> hand*, once for every platform we support (Windows, Linux x86, Android
>> NDK). Then we stuff the includes (headers) and libraries
>> (static/shared) in a submodule and the primary code base's CMake
>> scripts pull them in as interface targets.
>>
>> This works well and is light-weight but is a pain when upgrading or
>> changing libraries. It's a pain because if I want to upgrade boost, I
>> have to build it up to 6 times (once for each platform and once for
>> each configuration).
>>
>> I've been thinking of a different approach for a while. I've done some
>> toying around with the "Super Build" concept, where I have a separate
>> CMake project that does nothing but use the ExternalProject module to
>> build libraries in real time along with our project. So the order of
>> operations would be as follows (for our automated build server):
>>
>> 1. Clone our "Third Party" repository
>> 2. Use CMake to generate & build the "Super Build" project (this
>> builds boost, openssl, freetype, etc for the current platform).
>> 3. Clone the main code base's repository
>> 4. Use CMake to generate & build, using find_package() to refer to
>> interface targets exported by those third party libraries built in
>> step 2
>>
>> Obviously this will make builds take significantly longer, because
>> we're constantly rebuilding the same third party libraries over and
>> over again. However, it virtually eliminates the maintenance burden
>> for third party libraries because they are built inherently with
>> everything else.
>>
>> Note that I can't refer to pre-built libraries in our build
>> environments because we need very specific control over the versions
>> of our libraries as well as the toolchains that were used to build
>> them. Also we may specifically build our libraries a certain way (such
>> as boost). For this reason we do not rely on our external environment
>> or external package managers to fulfill third party dependencies, like
>> most open source projects do on Linux for example.
>>
>> Does this "Super Build" approach sound like a better idea? What other
>> options are available? The downside with the "Super Build" solution is
>> that it will become very difficult to make the transition between
>> building third party and building our code base seamless. I can't do
>> both in the same generate step because find_package() can't be called
>> until the libraries are built & installed.
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to 

Re: [CMake] Need ideas/opinions on third party library management

[Elizabeth A. Fischer]

>
> This is what Spack and other meta builders do.  I don't think CMake is the
> best place to do it, for a number of reasons.  I would not try to re-invent
> the wheel here.
>
See http://github. com/llnl/spack
>

-- Elizabeth
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Roger Leigh]

On 12/08/2016 19:59, Robert Dailey wrote:

Hello,



I've been thinking of a different approach for a while. I've done some
toying around with the "Super Build" concept, where I have a separate
CMake project that does nothing but use the ExternalProject module to
build libraries in real time along with our project. So the order of
operations would be as follows (for our automated build server):

1. Clone our "Third Party" repository
2. Use CMake to generate & build the "Super Build" project (this
builds boost, openssl, freetype, etc for the current platform).
3. Clone the main code base's repository
4. Use CMake to generate & build, using find_package() to refer to
interface targets exported by those third party libraries built in
step 2


You can simplify this to two steps:

1. Clone the superbuild repository
2. Build the third-party and first-party packages

This is the approach we take with the OME super-build
https://github.com/ome/ome-cmake-superbuild/

See the packages in
https://github.com/ome/ome-cmake-superbuild/tree/develop/packages
all the ones with an "ome-" prefix are the first-party libraries;
everything else is third-party.  They are all external projects.

We have cmake options to select all packages, first-party packages only 
or select individual packages by hand; the recursive dependency 
resolution will add/ignore dependencies depending upon the chosen policy 
(defaulting to include third-party dependencies).  So when you're 
building on BSD or Linux, you can say "no third-party dependencies 
except for gtest", or choose everything, and on Windows where there is 
no package manager, you always build everything.  And on Windows you can 
build just the dependencies and reuse them later; it has support for 
using cached builds which we use on the CI side to cut down build times; 
we only rebuild the cache when the superbuild git tree hash changes; see 
the scripts/jenkins-build script


In addition to that, we have the means to build the first-party 
libraries in different ways


1. From source release tarball (last stable release)
2. From git (specify remote+branch as cmake options)
3. From a local git checkout (on whatever branch is currently checked out)

You can see the selection logic here: 
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L4


In practice (1) is for end-users to build the whole thing, and (3) is 
what I use in practice to develop locally where I have a full set of git 
repos checked out locally, and I tell the superbuild to build them, 
along with all the third-party deps; (3) is also what we use on the CI 
system, with jenkins cloning all the individual repos into the 
workspace, then doing pretty much the same thing.  (2) is OK for one-off 
builds, but the need for a full git clone for every package is too much 
for routine use


And you can see how first- and third-party dependencies are handled 
here: 
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L48


I might rearrange that so that the classification is in the dependency 
rather than the dependeee later on though.  It kind of evolved this way 
as we added additional features and needs a little cleanup to make it 
perfect.


Everything in the above repository is BSD licensed, so feel free to use 
it for inspiration or take what you like.  The main point I wanted to 
make was that with the superbuild *everything* can be an external 
project, including your own code.  And from that point you can add in 
features like being able to selectively build just the third-party or 
just your own code, rather than forcing them to be separate manual steps.



Regards,
Roger
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Elizabeth A. Fischer]

This is what Spack and other meta builders do.  See http://github.
com/llnl/spack
On Aug 12, 2016 3:59 PM, "Robert Dailey"  wrote:

> Hello,
>
> There is an internal C++ product at the company I work for which I
> have written a series of CMake scripts for. This project actually has
> dependencies on several open source libraries, such as boost,
> freetype, openssl, etc.
>
> Right now what we do is build each of these third party libraries *by
> hand*, once for every platform we support (Windows, Linux x86, Android
> NDK). Then we stuff the includes (headers) and libraries
> (static/shared) in a submodule and the primary code base's CMake
> scripts pull them in as interface targets.
>
> This works well and is light-weight but is a pain when upgrading or
> changing libraries. It's a pain because if I want to upgrade boost, I
> have to build it up to 6 times (once for each platform and once for
> each configuration).
>
> I've been thinking of a different approach for a while. I've done some
> toying around with the "Super Build" concept, where I have a separate
> CMake project that does nothing but use the ExternalProject module to
> build libraries in real time along with our project. So the order of
> operations would be as follows (for our automated build server):
>
> 1. Clone our "Third Party" repository
> 2. Use CMake to generate & build the "Super Build" project (this
> builds boost, openssl, freetype, etc for the current platform).
> 3. Clone the main code base's repository
> 4. Use CMake to generate & build, using find_package() to refer to
> interface targets exported by those third party libraries built in
> step 2
>
> Obviously this will make builds take significantly longer, because
> we're constantly rebuilding the same third party libraries over and
> over again. However, it virtually eliminates the maintenance burden
> for third party libraries because they are built inherently with
> everything else.
>
> Note that I can't refer to pre-built libraries in our build
> environments because we need very specific control over the versions
> of our libraries as well as the toolchains that were used to build
> them. Also we may specifically build our libraries a certain way (such
> as boost). For this reason we do not rely on our external environment
> or external package managers to fulfill third party dependencies, like
> most open source projects do on Linux for example.
>
> Does this "Super Build" approach sound like a better idea? What other
> options are available? The downside with the "Super Build" solution is
> that it will become very difficult to make the transition between
> building third party and building our code base seamless. I can't do
> both in the same generate step because find_package() can't be called
> until the libraries are built & installed.
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Need ideas/opinions on third party library management

[Miller Henry]

A superbuild will work, and you can write your own Find*.cmake so that 
find_package works - you don't need most of what the real Find_package would do 
for you because you already know what is installed where as you control that.  
But, can you live with the trade off: long build times?

What my company did to solve a similar problem is write a tool (this took most 
of a year for a couple people, a lot that is very specific to our environment 
so sharing doesn't make sense) that that will install packages in a docker 
(linux only) image.  Each project we build has a "pin file" that lists the 
packages that project depends on.When we upgrade boost, the build creates a 
.rpm package which the tool automatically installs for the build to use.

What really makes our approach work is the integration into our CI system, when 
I change boost I verify the build in just one architecture, then push.  The CI 
system takes over to build every supported architecture, and then it deploys 
the packages - I don't have to build them all by hand (of course if something 
fails I need to figure that one out). Then I just have to pin the new package 
and our build system takes care of ensuring the new version is in place.

As I said, the above is complex and very custom to our environment.  I'm not 
entirely happy with it, but I can't come up with anything better for our 
requirements.  In particular I can go backward to any released version of code 
and rebuild it - meaning we can fix bugs in old versions easily if a customer 
isn't willing to upgrade.

-Original Message-
From: CMake [mailto:cmake-boun...@cmake.org] On Behalf Of Robert Dailey
Sent: Friday, August 12, 2016 3:00 PM
To: CMake ; CMake Developers 
Subject: [CMake] Need ideas/opinions on third party library management

Hello,

There is an internal C++ product at the company I work for which I have written 
a series of CMake scripts for. This project actually has dependencies on 
several open source libraries, such as boost, freetype, openssl, etc.

Right now what we do is build each of these third party libraries *by hand*, 
once for every platform we support (Windows, Linux x86, Android NDK). Then we 
stuff the includes (headers) and libraries
(static/shared) in a submodule and the primary code base's CMake scripts pull 
them in as interface targets.

This works well and is light-weight but is a pain when upgrading or changing 
libraries. It's a pain because if I want to upgrade boost, I have to build it 
up to 6 times (once for each platform and once for each configuration).

I've been thinking of a different approach for a while. I've done some toying 
around with the "Super Build" concept, where I have a separate CMake project 
that does nothing but use the ExternalProject module to build libraries in real 
time along with our project. So the order of operations would be as follows 
(for our automated build server):

1. Clone our "Third Party" repository
2. Use CMake to generate & build the "Super Build" project (this builds boost, 
openssl, freetype, etc for the current platform).
3. Clone the main code base's repository 4. Use CMake to generate & build, 
using find_package() to refer to interface targets exported by those third 
party libraries built in step 2

Obviously this will make builds take significantly longer, because we're 
constantly rebuilding the same third party libraries over and over again. 
However, it virtually eliminates the maintenance burden for third party 
libraries because they are built inherently with everything else.

Note that I can't refer to pre-built libraries in our build environments 
because we need very specific control over the versions of our libraries as 
well as the toolchains that were used to build them. Also we may specifically 
build our libraries a certain way (such as boost). For this reason we do not 
rely on our external environment or external package managers to fulfill third 
party dependencies, like most open source projects do on Linux for example.

Does this "Super Build" approach sound like a better idea? What other options 
are available? The downside with the "Super Build" solution is that it will 
become very difficult to make the transition between building third party and 
building our code base seamless. I can't do both in the same generate step 
because find_package() can't be called until the libraries are built & 
installed.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at