Which other projects depend on BigTop? How will the questions about the direction of BigTop affect those projects?
On Sun, Dec 7, 2014 at 6:10 PM, Roman Shaposhnik <[email protected]> wrote: > Hi! > > On Sat, Dec 6, 2014 at 3:23 PM, jay vyas <[email protected]> > wrote: > > hi bigtop ! > > > > I thought id start a thread a few vaguely related thoughts i have around > > next couple iterations of bigtop. > > I think in general I see two major ways for something like > Bigtop to evolve: > #1 remain a 'box of LEGO bricks' with very little opinion on > how these pieces need to be integrated > #2 start driving oppinioned use-cases for the particular kind of > bigdata workloads > > #1 is sort of what all of the Linux distros have been doing for > the majority of time they existed. #2 is close to what CentOS > is doing with SIGs. > > Honestly, given the size of our community so far and a total > lack of corporate backing (with a small exception of Cloudera > still paying for our EC2 time) I think #1 is all we can do. I'd > love to be wrong, though. > > > 1) Hive: How will bigtop to evolve to support it, now that it is much > more > > than a mapreduce query wrapper? > > I think Hive will remain a big part of Hadoop workloads for forseeable > future. What I'd love to see more of is rationalizing things like how > HCatalog, etc. need to be deployed. > > > 2) I wonder wether we should confirm cassandra interoperability of spark > in > > bigtop distros, > > Only if there's a significant interest from cassandra community and even > then my biggest fear is that with cassandra we're totally changing the > requirements for the underlying storage subsystem (nothing wrong with > that, its just that in Hadoop ecosystem everything assumes very HDFS'ish > requirements for the scale-out storage). > > > 4) in general, i think bigtop can move in one of 3 directions. > > > > EXPAND ? : Expanding to include new components, with just basic > interop, > > and let folks evolve their own stacks on top of bigtop on their own. > > > > CONTRACT+FOCUS ? Contracting to focus on a lean set of core > components, > > with super high quality. > > > > STAY THE COURSE ? Staying the same ~ a packaging platform for just > > hadoop's direct ecosystem. > > > > I am intrigued by the idea of A and B both have clear benefits and > costs... > > would like to see the opinions of folks --- do we lean in one direction > or > > another? What is the criteria for adding a new feature, package, stack to > > bigtop? > > > > ... Or maybe im just overthinking it and should be spending this time > > testing spark for 0.9 release.... > > I'd love to know what other think, but for 0.9 I'd rather stay the course. > > Thanks, > Roman. > > P.S. There are also market forces at play that may fundamentally change > the focus of what we're all working on in the year or so. >
