I guess it would totally make sense. Good you brought it up! Even if there are issues in 0.11 as Andrew mentioned - I imagine they might get fixed by the time of our release. Besides, Hive _always_ is full of problems: I dealt with its stinking guts for year and a half and am sure there's no way the problems will be fixed - short of rewriting a lot of runtime and query planning code. So, I won't worry as much ;)
Cos On Fri, Jul 12, 2013 at 01:05PM, Mark Grover wrote: > Also, what about Hive 0.11? > > It's been released for a while now. It's also the first release of Hive > that contains Hive Server2 (with support for concurrent queries). It also > contains HCatalog as a part of Hive (since HCatalog graduated the incubator > to become a part of Hive). I think this will be a great addition to Bigtop > 0.7. > > I understand, like Cos said, the focus of Bigtop 0.7 is to pay off our > technical debt but I think Hive 0.11 is worth the investment in Bigtop 0.7 > primarily because of all the goodness it brings in. > > If no one has any objections, can we add it to the BOM for 0.7 please? > > Thanks! > Mark > > On Thu, Jul 11, 2013 at 11:24 AM, Sean Mackrory <[email protected]>wrote: > > > I would agree. As I've said I really don't have a strong opinion on > > which one should be the "default" or whether we should do separate > > releases, but I also don't object to any of the proposals so far. I'm > > happy to lend a hand if needed, having already packaged these two > > side-by-side successfully before, but it's not very hard from a > > technical standpoint. > > > > > > > > On Wed, Jul 10, 2013 at 12:19 PM, Mark Grover <[email protected]> wrote: > > > Thanks everyone for your feedback! > > > > > > I think we are moving towards a consensus where we should put sqoop2 as > > the > > > only sqoop in Bigtop 0.7 BOM. We, as a community, are very open to adding > > > Sqoop1 back in Bigtop (0.6.1 or 0.7.0, whatever the decision is). We will > > > just have to ensure (and it shouldn't be too hard to do so) that there > > are > > > no namespace/command name conflicts between sqoop1 and the sqoop already > > > present in Bigtop (sqoop2). BIGTOP-1016 seems to be a good starting place > > > for that. > > > > > > Do folks agree with the above? > > > > > > Mark > > > > > > On Wed, Jul 10, 2013 at 10:52 AM, Konstantin Boudnik <[email protected]> > > wrote: > > > > > >> No, I meant stability of the framework itself: packaging, iTest, etc. > > >> Perhaps > > >> stability is too overloaded... robustness, perhaps? > > >> > > >> On Tue, Jul 09, 2013 at 10:57AM, Bruno Mahe wrote: > > >> > Bigtop as a framework? You mean stable api of its projects? > > >> > > > >> > Sent from my HTC EVO 4G LTE exclusively from Sprint > > >> > > > >> > ----- Reply message ----- > > >> > From: "Konstantin Boudnik" <[email protected]> > > >> > To: <[email protected]> > > >> > Cc: "Sean Mackrory" <[email protected]> > > >> > Subject: [DISCUSS] BOM for release 0.7.0 of Bigtop > > >> > Date: Tue, Jul 9, 2013 10:40 > > >> > > > >> > > > >> > Bruno, > > >> > > > >> > just to clarify my stance of 'stability': it is more about stability > > of > > >> the > > >> > Bigtop as a framework than a stability of the stack. > > >> > > > >> > I am not sure we have resources to do maintenance releases at this > > >> point. May > > >> > be it is just me. > > >> > > > >> > Cos > > >> > > > >> > On Tue, Jul 09, 2013 at 10:10AM, Bruno MahИ wrote: > > >> > > On 07/09/2013 09:47 AM, Sean Mackrory wrote: > > >> > >> Without wanting to detract from the spirit of focussing on system > > >> > >> stability, I'd like to suggest a few changes I think it's time we > > at > > >> least > > >> > >> discuss seriously: > > >> > >> > > >> > >> JDKs: I've seen a lot of people ask about JDK 7. Perhaps time to > > add > > >> > >> support for Oracle JDK 7? It's working pretty well in my > > experience, > > >> and > > >> > >> although it's less tested upstream, the only JDK we officially > > >> support is > > >> > >> officially EOL, so we're not exactly in a good position now IMO. > > >> > >> > > >> > >> Debian 7 has also been out for a while, and I think we should do at > > >> least > > >> > >> one release on it. It's likely very little work but I think there's > > >> value > > >> > >> in certifying the stack will work well there. (On the topic of > > OS's - > > >> are > > >> > >> we specifically talking SP3 of SLES 11?). I don't feel strongly on > > >> this, > > >> > >> but I'm just curious if there's a reason you're suggesting staying > > >> with > > >> > >> 12.10 and not 13.04 - other than wanting less change in this > > release? > > >> > >> Again, I hardly have an opinion on that one. > > >> > >> > > >> > >> Other components that have recently had releases that I don't > > >> consider to > > >> > >> impact the > > >> > >> - Hue 2.4.0 > > >> > >> - Whirr 0.8.2 > > >> > >> - Flume 1.4.0 > > >> > >> > > >> > >> There's also been a ticket to package Avro for a long time and I'd > > >> like to > > >> > >> get to that soon. Perhaps Parquet as well? Although like Phoenix > > and > > >> > >> DataFu, I would suggest doing just the libraries for now, not all > > the > > >> CLI > > >> > >> tools. > > >> > >> > > >> > >> Again - I don't mean to take away from the focus on stability, but > > I > > >> also > > >> > >> don't think we shouldn't stretch to stay up to date either. > > >> > >> > > >> > >> +1 to everything else as suggested, however. > > >> > >> > > >> > >> > > >> > >> On Tue, Jul 9, 2013 at 8:59 AM, Andrew Purtell < > > [email protected]> > > >> wrote: > > >> > >>> Would you be willing to consider Phoenix, only BIGTOP-993? > > >> Installing the > > >> > >>> package produced by 993 only drops a library for HBase into > > >> > >>> /usr/lib/phoenix, essentially the same relationship between the > > >> DataFu > > >> > >>> package and Pig. There is follow up work that is more ambitious, > > for > > >> > >>> example BIGTOP-1007, but that is not required by any means and > > could > > >> come > > >> > >>> in if/whenever you are comfortable with it. > > >> > >>> > > >> > >>> > > >> > >>> On Mon, Jul 8, 2013 at 8:50 PM, Konstantin Boudnik < > > [email protected]> > > >> wrote: > > >> > >>> > > >> > >>>> Guys, > > >> > >>>> > > >> > >>>> I wanna kick-off the discussion on the content of 0.7.0 BOM > > >> > >>>> Release 0.6.0 was all about stabilization of the stack and I > > think > > >> we > > >> > >> got a > > >> > >>>> great headway on that. The following components/OS were in the > > >> frame of > > >> > >> the > > >> > >>>> discussion: > > >> > >>>> http://is.gd/H52iVe > > >> > >>>> > > >> > >>>> My personal take that we need to spend this release cycle working > > >> on the > > >> > >>>> improvements in Bigtop itself: we got enough "technical debts" in > > >> the > > >> > >>>> pipeline that have to be addressed. To name a few: > > >> > >>>> - testability/test coverage > > >> > >>>> - test framework > > >> > >>>> - package improvements > > >> > >>>> - build improvements (including performance) > > >> > >>>> > > >> > >>>> In order to be able to deliver a solid stack again yet improve > > all > > >> things > > >> > >>>> Bigtop I'd like to focus on the latter, hence keeping the former > > at > > >> bay > > >> > >> and > > >> > >>>> limiting the component updates to the bugfix releases only (if > > >> > >> warranted). > > >> > >>>> E.g. > > >> > >>>> > > >> > >>>> Hadoop 2.0.5 or later (stabilization branch of Hadoop 2) > > >> > >>>> HBase 0.94.9 (update from the Bigtop 0.6.0) > > >> > >>>> HCatalog 0.5.0 (same as 0.6.0, as well as following...) > > >> > >>>> Zookeeper 3.4.5 > > >> > >>>> Pig 0.11.1 > > >> > >>>> Hive 0.10.0 > > >> > >>>> Sqoop 2 > > >> > >>>> Oozie 3.3.2 > > >> > >>>> Whirr 0.8.1 > > >> > >>>> Mahout 0.7 > > >> > >>>> Flume 1.3.1 > > >> > >>>> Giraph 0.2.0 > > >> > >>>> Hue 2.2.0 > > >> > >>>> Datafu 0.0.6 > > >> > >>>> Solr 4.2.1 > > >> > >>>> Crunch 0.5.0 > > >> > >>>> Tomcat 6.0.36 > > >> > >>>> Spark 0.7.3 (it has been in the queue for a long time) > > >> > >>>> > > >> > >>>> Also, I'd suggest to keep the same set of OSes as last time: > > >> > >>>> > > >> > >>>> CentOS/RHEL5 > > >> > >>>> CentOS/RHEL6 > > >> > >>>> SLES11 > > >> > >>>> Ubuntu 12.04 (LTS) > > >> > >>>> Fedora 18 > > >> > >>>> OpenSUSE 12.3 > > >> > >>>> Ubuntu 12.10 > > >> > >>>> > > >> > >>>> To reiterate, with a known stable version of the stack we can > > safely > > >> > >> focus > > >> > >>>> on > > >> > >>>> the improvements to the framework and the overall system > > usability. > > >> > >>>> > > >> > >>>> Please jump on the discussion Also, I have opened up the > > following > > >> JIRA > > >> > >> to > > >> > >>>> track the BOM update > > >> https://issues.apache.org/jira/browse/BIGTOP-1023 > > >> > >>>> > > >> > >>>> Thanks, > > >> > >>>> Cos > > >> > >>>> > > >> > >>>> > > >> > >>> > > >> > >>> > > >> > >>> -- > > >> > >>> Best regards, > > >> > >>> > > >> > >>> - Andy > > >> > >>> > > >> > >>> Problems worthy of attack prove their worth by hitting back. - > > Piet > > >> Hein > > >> > >>> (via Tom White) > > >> > >> > > >> > > > > >> > > > > >> > > +1 to what Andrew and Sean said. > > >> > > I would also like to add: > > >> > > * I would also like to keep the door open for Apache Gora to come > > in. > > >> > > Someone is already working on it > > >> > > * Fedora 19 is out and maybe 20 will be out by the time we release > > >> 0.7.0 > > >> > > > > >> > > With regard to stability, if included all the changes listed so far > > >> seem > > >> > > a little bit too much, what about dedicating a 0.6.1 to stability so > > >> > > 0.7.0 can include all the shiny things listed above? A 0.6.1 would > > >> > > enable us to be more strict with the patches/fixes/updates being > > >> > > included as well as signaling a more stable version to users. > > >> > > > > >> > > > > >> > > Thanks, > > >> > > Bruno > > >> > >
