On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yujie....@gmail.com> wrote:
> Kapil,
>
> I guess what I want to understand is why the existing structure makes it
> hard for you to do the things that you want to do (installing
> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as part
> of "make install").
Let me see if I can answer that :-).
This is somewhat related. For example, if we want to install protobuf
in 3rdparty/{include,lib} (for module developers to use them without
doing a proper mesos installation), you need to provide the correct
"--prefix" flag that points to 3rdparty/. However, due to multiple
levels of configure.ac, the "--prefix" can at best be generated by
prepending "../../../" to get to the great-grandparent directory. This
is because we have a separate configure.ac which manages
3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
but they are not clean.
Similar thing holds for system-wide installation of these 3rdparty
packages. For example, ideally, we would want to use
"${pkglibdir}/3rdparty" as a prefix for those packages. However, since
they are part of libprocess package, we don't get the correct
directory and have to use either hardwired $pkglibdir, or somehow pass
it from the top-level configure all the way down to
3rdparty/libprocess/3rdparty/Makefile.am :-(.
> The only reason you mentioned in the original email is that "in the current
> code base, we don't strictly follow the 3rdparty structure", which IMO is
> not a very convincing reason for such a big change.
How about a not so big change? :-). What if we just move
3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
stout as is? That is not a big change since we are not touching
libprocess/stout. Just adjusting Makefiles and I am pretty sure it
will be cleaner and simpler than what we have right now.
As a later time, we can then consider moving stout out to 3rdparty/
while leaving libprocess as is. But that's something we can decide
later and leave stout as an exception for now.
BTW, if we were to install all the 3rdparty packages in 3rdparty/,
that would also cut down a lot on the compiler flags (i.e., fewer "-I"
and "-L" flags) :-).
Kapil
>
> - Jie
>
> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>
>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yujie....@gmail.com> wrote:
>> >
>> > >
>> > > However, in the current code base, we don't strictly follow the
>> 3rdparty
>> > > structure. For example, stout has a dependency on picojson and
>> > > google-protobuf, but we don't put these two packages inside
>> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >
>> >
>> > My understanding is that stout is header only. So it does not have to
>> > bundle 3rdparty libraries. The user of stout is responsible for bundling
>> > them if they are used.
>>
>>
>> I don't think being header-only is an excuse to have a broken
>> installation :-). Further, we don't make it easier for the user to get
>> the 3rdparty binaries either. For example, if the user has a different
>> version of protobuf installed on the system, the compilation of any
>> program that uses stout will fail spectacularly!
>>
>> Having said that, the gist here is that we have somewhat deviated from
>> original motivation behind the 3rdparty directory and it would be nice
>> if we can have a flatter structure.
>>
>> >
>> >
>> > - Jie
>> >
>> > On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>> >
>> > > Hi All,
>> > >
>> > > TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>> 3rdparty/.
>> > > (Optionally) Move libprocess/stout to the top-level directory.
>> > >
>> > > I wanted to start some discussion around reorganizing stuff inside
>> > > "3rdparty". I apologize for the length of the email, please bear with
>> me.
>> > >
>> > > Traditionally, 3rdparty has been used to hold all Mesos dependencies
>> > > (zookeeper, libprocess, protobuf, stout, etc.). Further,
>> > > libprocess/3rdparty was to hold all libprocess dependencies (which may
>> in
>> > > turn be Mesos dependencies as well).
>> > >
>> > > As I understand, the original motivation was to emphasize that
>> libprocess
>> > > is an independent project which depends on "stout", which in turn is
>> also
>> > > an independent project.
>> > >
>> > > However, in the current code base, we don't strictly follow the
>> 3rdparty
>> > > structure. For example, stout has a dependency on picojson and
>> > > google-protobuf, but we don't put these two packages inside
>> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> > >
>> > > In light of these anomalies, I want to propose that we flatten out the
>> > > 3rdparty directory and put all packages (libprocess, stout, protobuf,
>> > > picojson, zookeeper, etc.) at the same level in 3rdparty. We can still
>> use
>> > > "--with-XYZ=..." to the full extent as needed.
>> > >
>> > > To take it a step further, I want to propose that we bring libprocess
>> and
>> > > stout out of 3rdparty/ and move them at the top level (i.e., make them
>> > > peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>> > > parties and is used only when "--with-bundled" is defined (by default).
>> > > This hierarchy will still allow us to keep libprocess and stout as
>> separate
>> > > independent projects.
>> > >
>> > > The motivation for this proposal came when dealing with 3rdparty
>> > > dependencies for module development. A module developer needs access to
>> > > protobuf, picojson, glog, etc., and for that matter, the exact
>> versions of
>> > > these packages that Mesos was compiled with.
>> > >
>> > > We want to solve this problem by installing module-specific 3rdparty
>> > > dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
>> > > configured with something like "--enable-install-module-dependencies").
>> > > (There is a discussion going on in a separate thread).
>> > >
>> > > Further, as of today, when we install Mesos using "make install", we
>> > > install stout headers in "${prefix}/include/stout". However, stout has
>> > > dependencies on picojson[1] and google-protobuf headers which may not
>> be
>> > > present on the machine. This leaves stout, and in turn libprocess and
>> Mesos
>> > > headers, fairly broken. I understand that this issue is somewhat
>> orthogonal
>> > > to the main issue being discussed in this mail, but I wanted to put it
>> out
>> > > since it's related.
>> > >
>> > > Any thoughts, comments, concerns are most welcome!
>> > >
>> > > Best,
>> > > Kapil
>> > >
>> > >
>> > > [1]: Picojson issue was resolved as part of
>> > > https://reviews.apache.org/r/41424/ which installs picojson.h into the
>> > > include-dir.
>> > >
>>
