Thanks Jay for starting this discussion. I'm interested in this topic. Actually my knowledge on k8s is limited but, my colleague is working on related on this. It would be nice to keep on discussion to shape up the blueprint and future.
Thanks, Youngwoo On Thu, Mar 9, 2017 at 5:42 AM, jay vyas <[email protected]> wrote: > Hi bigtop: > > Since i work on kubernetes now adays, every once in a while I like to drop > an email to let you guys know what I've been up to, and how bigtop might > play a role in the new containerized world we live in. > > These are just some ideas to cross fertilize the community, not official > bigtop proposals, although I'm happy to discuss this further if anyone is > interested in running bigtop components in a containerized data cetner > environment. > > I think if anyone ever wanted to run bigtop in a containerized way, it > might be interesting to select components we care about, and create a > version of bigtop that deployed as "cloud native". Rather then > pontificating on merits, ill just discuss 2 ways this might look like if it > ever was done: > > 1) Deployment as an idiomatic containerized service. > > In order to do this, we basically could do something like: > > - Publish yaml files as part of a bigtop release. > - Define stateful sets for the things we care about, like spark masters, or > ignite, or your hadoop nodes. > - Deploy and scale up/down as needed. > > In kube, this is done via PetSets for performance or statefulness, but WLOG > the above instructions could possibly work in other places, i.e. like > docker swarm or something. > > 2) Deployment by simulating "nodes" (anti-pattern but might be a good > gateway to accomodate migration. > > Another way to do this same thing might be to simply deploy a set of JVM > containers in a cluster, which could serve as a "bigtop base image" (we do > this now with the bigtop tool chain, sorta), in fact, we already publish > containers for this iirc. This would break the containerized idiom that > is most popular (microservices), but might be an interesting staging ground > for people to deploy static hadoop clusters in a containerized microsystem > environment. > > -- > jay vyas >
