We obviously need to get Impala to work with non-CDH hadoop components, but it seems sane to me that we support only specific versions of the native dependencies.
I think the native-toolchain maybe needs some more thought - particularly downloading source and binaries from S3, but if you were determined you could build all the dependencies independently from source, put them in a toolchain directory, then build Impala. On Thu, May 26, 2016 at 8:32 AM, Jim Apple <[email protected]> wrote: > I agree. I also think that it's a shame Impala depends so precariously > on the exact versions of various packages. It is especially concerning > to me when those are CDH packages, or when those packages are > maintained by cloudera on cloudera's github account. > > On Wed, May 25, 2016 at 5:22 PM, Michael Ho <[email protected]> wrote: > > Other than the logistics reasons Tim mentioned, my thinking is that > > thirdparty and toolchain overlap a lot so it would be easier to have just > > one location for everything. > > > > Thanks, > > Michael > > > > On May 25, 2016, at 16:46, Henry Robinson <[email protected]> wrote: > > > > On 25 May 2016 at 16:33, Michael Ho <[email protected]> wrote: > >> > >> Yes. The biggest dependency in thirdparty now is the reliance of the CDH > >> components, namely > >> hadoop, hbase, hive, sentry and some llama stuff. The short term goal as > >> discussed in another email > >> thread is to initially put a publicly available released version of > those > >> CDH components in S3 so a user > >> can get all the components they need to build and run tests by just > >> running the script to bootstrap the > >> toolchain. Internally, we can replace those CDH components with our own > >> builds of those components > >> by somehow instructing the toolchain boostrapping script to pull from > our > >> internal repo if needed. The > >> users can in theory specify their own repo for those components too. > >> Whether we will converge to > >> the Apache version of things in the long run is not the goal of this > bug. > >> > >> The majority of the stuff left in thirdparty are mostly duplicates of > what > >> exist already in > >> the toolchain directory: avro-c, sasl, gflags, glog, gperftools, gtest, > >> lz4, openldap, rapidjson, re2, snappy, > >> thrift so we may as well remove them. > >> > >> After all the changes above, thirdparty will be pretty much empty except > >> for mustache and squeasel. > >> If I recalled correctly from the meeting we had a month ago before you > >> went on leave, you pointed out > >> that they can reside in util directory as they are actually compiled > >> together with the webserver.cc. I am > >> also open to keeping them in thirdparty directory as this means we can > now > >> have a thirdparty directory > >> in the ASF repo upstream after all the trimming above. > > > > > > Thanks for the explanation. I still don't see from your response why you > > think Squeasel and Mustache should move to util/, even if, as we > discussed, > > they can (there are good reasons to consider moving them - maybe we don't > > want thirdparty/ any more, maybe we don't want to treat external compile > > dependencies any differently). > > > > The argument I can see in favour of not moving them is that any external > > compile-time dependencies are in an easy-to-find place, which probably > makes > > getting an Apache release together just a little easier since we can > split > > the code into "developed as part of Impala" and "developed externally". > > > > I'm not going to stand in the way of a small change like this, so feel > free > > to GVM if you think best. > > > >> > >> > >> Michael > >> > >> On Wed, May 25, 2016 at 3:30 PM, Henry Robinson <[email protected]> > wrote: > >>> > >>> I thought the long-term goal was to remove any heavyweight dependencies > >>> that are checked into the repository. It's not the existence of > thirdparty/ > >>> that's the problem there, it's what we were keeping in it. > >>> > >>> Mustache and Squeasel change very infrequently, and are only a few K in > >>> total. Do you think we should remove thirdparty/ for other reasons? > >>> > >>> On 25 May 2016 at 14:20, Michael Ho <[email protected]> wrote: > >>>> > >>>> The long term goal is to remove thirdparty. I thought we discussed > about > >>>> it. > >>>> > >>>> On Wed, May 25, 2016 at 2:13 PM, Henry Robinson <[email protected]> > >>>> wrote: > >>>>> > >>>>> (Sorry, not near Gerrit) why not keep them in thirdparty/? That makes > >>>>> their provenance clear. > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>> > On May 25, 2016, at 2:06 PM, Dan Hecht (Code Review) > >>>>> > <[email protected]> wrote: > >>>>> > > >>>>> > Dan Hecht has posted comments on this change. > >>>>> > > >>>>> > Change subject: IMPALA-3223: Move squeasel and mustache to the util > >>>>> > directory > >>>>> > > >>>>> > > ...................................................................... > >>>>> > > >>>>> > > >>>>> > Patch Set 1: Code-Review+2 > >>>>> > > >>>>> > Okay. > >>>>> > > >>>>> > -- > >>>>> > To view, visit http://gerrit.cloudera.org:8080/3206 > >>>>> > To unsubscribe, visit http://gerrit.cloudera.org:8080/settings > >>>>> > > >>>>> > Gerrit-MessageType: comment > >>>>> > Gerrit-Change-Id: I782e493bc28086a1587274b3c474ea6b6f201855 > >>>>> > Gerrit-PatchSet: 1 > >>>>> > Gerrit-Project: Impala > >>>>> > Gerrit-Branch: cdh5-trunk > >>>>> > Gerrit-Owner: Michael Ho <[email protected]> > >>>>> > Gerrit-Reviewer: Dan Hecht <[email protected]> > >>>>> > Gerrit-Reviewer: Jim Apple <[email protected]> > >>>>> > Gerrit-Reviewer: Michael Ho <[email protected]> > >>>>> > Gerrit-HasComments: No > >>>>> > > >>>>> > -- > >>>>> > You received this message because you are subscribed to the Google > >>>>> > Groups "impala-cr" group. > >>>>> > To unsubscribe from this group and stop receiving emails from it, > >>>>> > send an email to [email protected]. > >>>>> > For more options, visit > >>>>> > https://groups.google.com/a/cloudera.com/d/optout. > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Thanks, > >>>> Michael > >>> > >>> > >> > >> > >> > >> -- > >> Thanks, > >> Michael > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "impala-cr" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected]. > >> For more options, visit > https://groups.google.com/a/cloudera.com/d/optout. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "impala-cr" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/a/cloudera.com/d/optout. >
