> Is it time for us to pick a resource manager? Not if we want to be like a Debian for big data software. I'm not sure we want to limit our reach by being overly opinionated. With my user's hat on, if we don't package Hadoop and YARN, then I wouldn't have any use for Bigtop.
> Nowadays folks don't necessarily need the whole hadoop ecosystem, and there is a huge shift to in-memory, monolithic stacks happening A Bigtop user would only need to install the packages they would like to use, right? Is this an argument for exclusion? Exclusion of what? > Is it time to start experimenting with running docker tarballs ? This sounds fine as an additional target for builds, but not if it leads to a proposal to do away with the OS native packaging. That's useful too. Containers are trendy but not useful or even appropriate for every environment or use case. On Mon, Jun 15, 2015 at 9:22 AM, jay vyas <[email protected]> wrote: > Hi folks. Every few months, i try to reboot the conversation about the > next generation of bigtop. > > There are 3 things which i think we should consider : A backplane (rather > than deploy to machines, the meaning of the term "ecosystem" in a > post-spark in-memory apacolypse, and containerization. > > 1) BACKPLANE: The new trend is to have a backplane that provides > networking abstractions for you (mesos, kubernetes, yarn, and so on). Is > it time for us to pick a resource manager? > > 2) ECOSYSTEM?: Nowadays folks don't necessarily need the whole hadoop > ecosystem, and there is a huge shift to in-memory, monolithic stacks > happening (i.e. gridgain or spark can do what 90% of the hadoop ecosystem > already does, supporting streams, batch,sql all in one). > > 3) CONTAINERS: we are doing a great job w/ docker in our build infra. Is > it time to start experimenting with running docker tarballs ? > > Combining 1+2+3 - i could see a useful bigdata upstream distro which (1) > just installed an HCFS implementation (gluster,HDFS,...) along side, say, > (2) mesos as a backplane for the tooling for [[ hbase + spark + ignite ]] > --- and then (3) do the integration testing of available mesos-framework > plugins for ignite and spark underneath. If other folks are interested, > maybe we could create the "1x" or "in-memory" branch to start hacking on it > sometime ? Maybe even bring the flink guys in as well, as they are > interested in bigtop packaging. > > > > -- > jay vyas > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
