This is a great time to discuss the Mesos dependency channel story in general, because it has had to evolve a bit to fit the requirements of Windows, and some of the issues you describe are issues we had to resolve (at least partially) to support the Windows integration work.
More particularly, our problems are: first, Windows frequently requires newer versions of dependencies (due to poor support of MSVC 1900), so we have had to develop reasonably robust version-selection mechanisms, so that Windows can get specific versions of different packages. This means that the Mesos project does not have to evolve the dependency support story in lock step, which in the long term may actually be required, as some platforms (e.g., those run by governmental organizations) are more conservative about what dependencies are introduced on their clusters. Second, because Windows does not have a package manager, it became necessary for the CMake build system to support actually hitting some remote (possiblty the internet) to rope in the tarballs of arbitrary (and arbitrarily-versioned) dependencies that we normally expect to already be installed (such as APR or cURL). This last point is actually more convenient than it seems. Our CMake implementation recently[1][2] introduced a flag that lets you specify something like `cmake .. -D3RDPARTY_DEPENDENCIES=/some/path/or/url` and it will proactively look for tarballs in the location you give it -- and that location can be either a path on your filesystem, or a URI, like the 3rdparty remote in github[3] that is owned by the GitHub community. From the "exotic platform" perspective this is great because it makes it trivial for people building (say) Windows to upgrade to a version not supported by CMake: * Put a tarball of a new version somewhere on the filesystem. Say, we decide to use glog 0.3.4 instead of 0.3.3, so we just put that tarball for 0.3.4 in a well-known place in the filesystem. * Update the version of glog in Versions.cmake. * When you run cmake, just run `cmake .. -D3RDPARTY_DEPENDENCIES=/my/fancy/3rdparty/path` * Builds against new dep! Magic! Much of this was developed out of expediency, but going forward I think a reasonable approach to dealing with the third-party channel might be (and I would LOVE feedback on this): WORKFLOW THAT ASSUMES INTERNET ACCESS ON BUILD MACHINE: * Clone a copy of mesos. * (When we do a normal clone of Mesos, there are no tarballs in the `3rdparty/` directory.) * Run `bootstrap`. * `mkdir build && cd build && cmake ..`. Part of the CMake configuration step will be to `git clone` a copy of `https://github.com/3rdparty/mesos-3rdparty`. (If you don't know, the 3rdparty account is owned by the Mesos community, and the `mesos-3rdparty` is where we store canonical copies of all our third-party tarballs.) * This dumps all the tarballs into a folder, `mesos-3rdparty`. * We build against the tarballs we retrieved. Optionally you are allowed to set the versions in `Versions.cmake` and mesos will "just build" against those versions (as long as they are supported, and we will complain if they're not). * If you `git pull` and find that Mesos has upgraded its dependencies, and a version is out of date, then when you next build, CMake will explode automatically (even if you've built before) and ask you to `git pull` to update your `mesos-3rdparty` repository. WORKFLOW THAT DOES NOT ASSUME INTERNET ACCESS ON BUILD MACHINE: Much like the above, except when you run cmake, you do `cmake .. -D3RDPARTY_DEPENDENCIES="path/to/mesos-3rdparty/mirror"`. This will tell CMake to not clone the mirror itself, but to look for an existing mirror at the location specified. WHAT WE'VE IMPLEMENTED: We obviously haven't deleted the tarballs in 3rdparty, and the error reporting around `Versions.cmake` and asking people to re-pull when a version has been upgraded are not there, but a lot of the rest of this is already in place. For example, yesterday we checked in an implementation of the `-D3RDPARTY_DEPENDENCIES` flag[1][2], which allows you to tell CMake to build against third-party dependencies mirrored either at a local path (e.g., `-D3RDPARTY_DEPENDENCIES="/your/path/here"`) or at a remote URI (e.g., `-D3RDPARTY_DEPENDENCIES=https://github.com/3rdparty/mesos-3rdparty`). [1] https://github.com/apache/mesos/commit/6306b7d62dd5cbb34fa82636dfbb46cee46d0bf8 [2] https://github.com/apache/mesos/commit/3f7501b818662097f41b2d756b2389f6ed9fa5eb [3] https://github.com/3rdparty/mesos-3rdparty On Tue, Mar 1, 2016 at 7:56 AM, Kapil Arya <[email protected]> wrote: >> >> *3. 3rdparty/libprocess/3rdparty/stout/tests/protobuf_tests.pb.cc/h >> <http://protobuf_tests.pb.cc/h> files.* >> >> Can anyone tell me why hardcode these two files in Mesos repo? I think >> these two files can be dynamically generated during make check, this will >> make it not depend on protoc version. >> > > I think it's just due to the nature of the way dependencies are structured > in 3rdparty. Alex Rukletsov and I thought about fixing it but at that time, > there was some complication due to protoc related dependency paths not > being resolved properly or something like that (I don't remember exactly). > I think there is a way to do it in the current structure, but I strongly > suspect that this will get much better if/when we go ahead with 3rdparty > flattening. > > It will be great if you have any other comments, thanks. >> -- Alex Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)
