If a user upgrades a dependency, he only need to update the 3rdparty/libray_name directory with his updates, no need to re-run the bootstrap script.
After the user upgrade a dependency, he need to re-run `make` command, and make command can catch up this new change. On Tue, Mar 8, 2016 at 1:58 PM, Alex Clemmer <clemmer.alexan...@gmail.com> wrote: > It's not clear to me what happens when a user upgrades a dependency. > It seems like they'd need to re-run the `bootstrap` script, is that > correct? > > On Mon, Mar 7, 2016 at 5:23 PM, zhiwei <zhiw...@gmail.com> wrote: > > I don't think store tarballs in Mesos git repository is a good idea, even > > in another 3rdparty repo. > > > > *So my opinion is:* > > > > Add a 3rdparty configure file(/support/3rdparty.config), the file format > > can be: > > > > library_name git_repo_url git_tag/branch/commit_id .patch_file > > > > zookeeper https://github.com/apache/zookeeper.git release-3.4.8 > > zookeeper.patch > > leveldb https://github.com/google/leveldb.git v1.4 leveldb.patch > > ... ... ... ... > > > > In bootstrap file: > > > > 1. Traverse the support/3rdparty.config > > > > 2. If there is already a 3rdparty/library_name directory, skip and > continue > > another item. > > > > 3. Clone the library code, switch to the defined tag/branch/commit_id, > > apply the .patch file. > > > > 4. Do the rest steps. > > > > So if users want to use their own 3rdparty libraries, he can checkout > code > > to the /3rdparty/library_name and the bootstrap script will skip this > > library. > > > > And in Mesos repository, we only need to maintain the > > support/3rdparty.config file and the .patch files. > > > > > > On Tue, Mar 8, 2016 at 2:05 AM, Alex Clemmer < > clemmer.alexan...@gmail.com> > > wrote: > > > >> So, at this point we have a bunch of different reviews open for > >> this[1], and I'd like to use this as an opportunity to start nudging > >> people towards thinking about possibly transitioning to a scheme where > >> the tarballs that are currently held in the `3rdparty/` directory are > >> moved to some external place, and retrieved for users out-of-band, as > >> necessary to build Mesos. > >> > >> In particular, doing this (as you all likely know) is very expensive > >> because git stores a complete copy of the entire tarball, for each > >> different revision in history, so if you have updated a tarball twice, > >> you have two complete copies rolling around in the `.git/` folder. It > >> seems like there are not many benefits for keeping this scheme, other > >> than the fact that it's pretty easy to implement. > >> > >> I'm not sure what it would take to transition the autotools build > >> system, but just recapping earlier what I said about the CMake build > >> system: The easiest thing to do (which we've already mostly done) is > >> to allow people to rope in tarballs from some mirror of the `3rdparty` > >> github repository[2]. Right now we have facilities that let you host > >> it either on your local FS or on a remote URL, and we'll download (if > >> necessary) and untar into the familliar place in the `build/` folder. > >> Easy! We could even have `bootstrap` clone the repository and make > >> CMake automatically pull in that repository if it's out of date. > >> > >> Thoughts? I recognize that this might be overcomplicating the problem > >> a bit, but I figured I'd throw the hat in the ring because this has > >> always kind of bothered me. :) > >> > >> > >> [1] They are: > >> https://reviews.apache.org/r/44252/ > >> https://reviews.apache.org/r/44382/ > >> https://reviews.apache.org/r/44372/ > >> https://reviews.apache.org/r/44378/ > >> https://reviews.apache.org/r/44376/ > >> https://reviews.apache.org/r/44257/ > >> > >> [2] https://github.com/3rdparty/mesos-3rdparty > >> > >> On Tue, Mar 1, 2016 at 10:48 AM, Alex Clemmer > >> <clemmer.alexan...@gmail.com> wrote: > >> > It doesn't seem to be the case that these things are mutually > >> > exclusive -- it is well within our purview to accept only a specific > >> > range of versions for any particular dependency, and error out if > >> > someone tries to select a version outside that range. The only thing > >> > these commits add is more fine-grained control over which of the > >> > supported versions you are allowed to select. > >> > > >> > At this point, there are no such guards, but that is certainly > >> > something that can be added. > >> > > >> > On Tue, Mar 1, 2016 at 10:17 AM, Neil Conway <neil.con...@gmail.com> > >> wrote: > >> >> The prospect of downloading dependencies from "rando" locations is > >> >> concerning to me :) > >> >> > >> >> Mesos can easily come to depend on implementation details of a > >> >> dependency that might change in a minor release. For example, a > recent > >> >> change [1] depends on the connection retry logic in the Zk client > >> >> library in a fairly delicate way. I also wouldn't want users to > >> >> randomly upgrade to, say, protobuf 2.6.1 without it being thoroughly > >> >> tested. Increasing the support matrix of different users on different > >> >> platforms running arbitrarily different versions of third-party > >> >> dependencies doesn't seem like a net improvement to me. > >> >> > >> >> My two cents: if Windows requires additional dependencies that we > >> >> aren't currently vendoring, I would personally opt for (a) vendoring > >> >> those additional dependencies (b) ensuring that the vendored versions > >> >> we ship are modern enough to support all the platforms we care about. > >> >> Are there important use-cases that aren't supported by this scheme? > >> >> > >> >> Neil > >> >> > >> >> [1] > >> > https://github.com/apache/mesos/commit/c2d496ed430eaf7daee3e57edefa39c25af2aa43 > >> >> > >> >> On Tue, Mar 1, 2016 at 10:00 AM, Alex Clemmer > >> >> <clemmer.alexan...@gmail.com> wrote: > >> >>> I guess a tl;dr might be in order. > >> >>> > >> >>> Basically: the CMake build system already supports roping in > tarballs > >> >>> from rando places on the filesystem or Internet, so I think it makes > >> >>> sense to rope them in at configure time, and so I'm proposing we > >> >>> re-appropriate the sophisticated tools we already have to do this > for > >> >>> WIndows, into a more general solution that is useful to other exotic > >> >>> platforms, rather than just Windows. > >> >>> > >> >>> As always, super interested to hear feedback, I'd love to know if I > >> >>> missed something. > >> >>> > >> >>> On Tue, Mar 1, 2016 at 9:58 AM, Alex Clemmer > >> >>> <clemmer.alexan...@gmail.com> wrote: > >> >>>> This is a great time to discuss the Mesos dependency channel story > in > >> >>>> general, because it has had to evolve a bit to fit the > requirements of > >> >>>> Windows, and some of the issues you describe are issues we had to > >> >>>> resolve (at least partially) to support the Windows integration > work. > >> >>>> > >> >>>> More particularly, our problems are: first, Windows frequently > >> >>>> requires newer versions of dependencies (due to poor support of > MSVC > >> >>>> 1900), so we have had to develop reasonably robust > version-selection > >> >>>> mechanisms, so that Windows can get specific versions of different > >> >>>> packages. This means that the Mesos project does not have to evolve > >> >>>> the dependency support story in lock step, which in the long term > may > >> >>>> actually be required, as some platforms (e.g., those run by > >> >>>> governmental organizations) are more conservative about what > >> >>>> dependencies are introduced on their clusters. > >> >>>> > >> >>>> Second, because Windows does not have a package manager, it became > >> >>>> necessary for the CMake build system to support actually hitting > some > >> >>>> remote (possiblty the internet) to rope in the tarballs of > arbitrary > >> >>>> (and arbitrarily-versioned) dependencies that we normally expect to > >> >>>> already be installed (such as APR or cURL). > >> >>>> > >> >>>> This last point is actually more convenient than it seems. Our > CMake > >> >>>> implementation recently[1][2] introduced a flag that lets you > specify > >> >>>> something like `cmake .. -D3RDPARTY_DEPENDENCIES=/some/path/or/url` > >> >>>> and it will proactively look for tarballs in the location you give > it > >> >>>> -- and that location can be either a path on your filesystem, or a > >> >>>> URI, like the 3rdparty remote in github[3] that is owned by the > GitHub > >> >>>> community. From the "exotic platform" perspective this is great > >> >>>> because it makes it trivial for people building (say) Windows to > >> >>>> upgrade to a version not supported by CMake: > >> >>>> > >> >>>> * Put a tarball of a new version somewhere on the filesystem. Say, > we > >> >>>> decide to use glog 0.3.4 instead of 0.3.3, so we just put that > tarball > >> >>>> for 0.3.4 in a well-known place in the filesystem. > >> >>>> * Update the version of glog in Versions.cmake. > >> >>>> * When you run cmake, just run `cmake .. > >> >>>> -D3RDPARTY_DEPENDENCIES=/my/fancy/3rdparty/path` > >> >>>> * Builds against new dep! Magic! > >> >>>> > >> >>>> Much of this was developed out of expediency, but going forward I > >> >>>> think a reasonable approach to dealing with the third-party channel > >> >>>> might be (and I would LOVE feedback on this): > >> >>>> > >> >>>> WORKFLOW THAT ASSUMES INTERNET ACCESS ON BUILD MACHINE: > >> >>>> * Clone a copy of mesos. > >> >>>> * (When we do a normal clone of Mesos, there are no tarballs in the > >> >>>> `3rdparty/` directory.) > >> >>>> * Run `bootstrap`. > >> >>>> * `mkdir build && cd build && cmake ..`. Part of the CMake > >> >>>> configuration step will be to `git clone` a copy of > >> >>>> `https://github.com/3rdparty/mesos-3rdparty`. (If you don't know, > the > >> >>>> 3rdparty account is owned by the Mesos community, and the > >> >>>> `mesos-3rdparty` is where we store canonical copies of all our > >> >>>> third-party tarballs.) > >> >>>> * This dumps all the tarballs into a folder, `mesos-3rdparty`. > >> >>>> * We build against the tarballs we retrieved. Optionally you are > >> >>>> allowed to set the versions in `Versions.cmake` and mesos will > "just > >> >>>> build" against those versions (as long as they are supported, and > we > >> >>>> will complain if they're not). > >> >>>> * If you `git pull` and find that Mesos has upgraded its > dependencies, > >> >>>> and a version is out of date, then when you next build, CMake will > >> >>>> explode automatically (even if you've built before) and ask you to > >> >>>> `git pull` to update your `mesos-3rdparty` repository. > >> >>>> > >> >>>> > >> >>>> WORKFLOW THAT DOES NOT ASSUME INTERNET ACCESS ON BUILD MACHINE: > >> >>>> Much like the above, except when you run cmake, you do `cmake .. > >> >>>> -D3RDPARTY_DEPENDENCIES="path/to/mesos-3rdparty/mirror"`. This will > >> >>>> tell CMake to not clone the mirror itself, but to look for an > existing > >> >>>> mirror at the location specified. > >> >>>> > >> >>>> > >> >>>> WHAT WE'VE IMPLEMENTED: > >> >>>> We obviously haven't deleted the tarballs in 3rdparty, and the > error > >> >>>> reporting around `Versions.cmake` and asking people to re-pull > when a > >> >>>> version has been upgraded are not there, but a lot of the rest of > this > >> >>>> is already in place. For example, yesterday we checked in an > >> >>>> implementation of the `-D3RDPARTY_DEPENDENCIES` flag[1][2], which > >> >>>> allows you to tell CMake to build against third-party dependencies > >> >>>> mirrored either at a local path (e.g., > >> >>>> `-D3RDPARTY_DEPENDENCIES="/your/path/here"`) or at a remote URI > (e.g., > >> >>>> `-D3RDPARTY_DEPENDENCIES= > https://github.com/3rdparty/mesos-3rdparty` > >> ). > >> >>>> > >> >>>> > >> >>>> [1] > >> > https://github.com/apache/mesos/commit/6306b7d62dd5cbb34fa82636dfbb46cee46d0bf8 > >> >>>> [2] > >> > https://github.com/apache/mesos/commit/3f7501b818662097f41b2d756b2389f6ed9fa5eb > >> >>>> [3] https://github.com/3rdparty/mesos-3rdparty > >> >>>> > >> >>>> On Tue, Mar 1, 2016 at 7:56 AM, Kapil Arya <ka...@mesosphere.io> > >> wrote: > >> >>>>>> > >> >>>>>> *3. 3rdparty/libprocess/3rdparty/stout/tests/ > protobuf_tests.pb.cc/h > >> >>>>>> <http://protobuf_tests.pb.cc/h> files.* > >> >>>>>> > >> >>>>>> Can anyone tell me why hardcode these two files in Mesos > repo? > >> I think > >> >>>>>> these two files can be dynamically generated during make check, > >> this will > >> >>>>>> make it not depend on protoc version. > >> >>>>>> > >> >>>>> > >> >>>>> I think it's just due to the nature of the way dependencies are > >> structured > >> >>>>> in 3rdparty. Alex Rukletsov and I thought about fixing it but at > >> that time, > >> >>>>> there was some complication due to protoc related dependency paths > >> not > >> >>>>> being resolved properly or something like that (I don't remember > >> exactly). > >> >>>>> I think there is a way to do it in the current structure, but I > >> strongly > >> >>>>> suspect that this will get much better if/when we go ahead with > >> 3rdparty > >> >>>>> flattening. > >> >>>>> > >> >>>>> It will be great if you have any other comments, thanks. > >> >>>>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> Alex > >> >>>> > >> >>>> Theory is the first term in the Taylor series of practice. -- > Thomas M > >> >>>> Cover (1992) > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> Alex > >> >>> > >> >>> Theory is the first term in the Taylor series of practice. -- > Thomas M > >> >>> Cover (1992) > >> > > >> > > >> > > >> > -- > >> > Alex > >> > > >> > Theory is the first term in the Taylor series of practice. -- Thomas M > >> > Cover (1992) > >> > >> > >> > >> -- > >> Alex > >> > >> Theory is the first term in the Taylor series of practice. -- Thomas M > >> Cover (1992) > >> > > > > -- > Alex > > Theory is the first term in the Taylor series of practice. -- Thomas M > Cover (1992) >
