I would love to see this happening. But I guess folks here does not know k8s that well as you do.
Could you draft plans like how we're going to make progress? For example, I'm just random guessing - the short-term plan is to wrap a script to put bigtop on k8s using existing deployment solution(puppet), the long-term plan is to leverage something that k8s naively support for deployment. What's your suggestion, jay? Evans 2017-03-11 10:28 GMT+08:00 Roman Shaposhnik <[email protected]>: > On Wed, Mar 8, 2017 at 12:42 PM, jay vyas <[email protected]> > wrote: > > Hi bigtop: > > > > Since i work on kubernetes now adays, every once in a while I like to > drop > > an email to let you guys know what I've been up to, and how bigtop might > > play a role in the new containerized world we live in. > > > > These are just some ideas to cross fertilize the community, not official > > bigtop proposals, although I'm happy to discuss this further if anyone is > > interested in running bigtop components in a containerized data cetner > > environment. > > I'm very curious where this could lead us. Interestingly enough, I've > offered > Google and CNCF similar ideas for collaboration at ODPi level, but those > guys didn't really bite. > > > I think if anyone ever wanted to run bigtop in a containerized way, it > > might be interesting to select components we care about, and create a > > version of bigtop that deployed as "cloud native". Rather then > > pontificating on merits, ill just discuss 2 ways this might look like if > it > > ever was done: > > > > 1) Deployment as an idiomatic containerized service. > > > > In order to do this, we basically could do something like: > > > > - Publish yaml files as part of a bigtop release. > > - Define stateful sets for the things we care about, like spark masters, > or > > ignite, or your hadoop nodes. > > - Deploy and scale up/down as needed. > > > > In kube, this is done via PetSets for performance or statefulness, but > WLOG > > the above instructions could possibly work in other places, i.e. like > > docker swarm or something. > > Interesting! I've followed Kubernetes development somewhat, but at a pretty > high level. What's a good intro (presentation, etc.) into those latest > stateful > capabilities and what's a good way to play with those hands-on? > > > 2) Deployment by simulating "nodes" (anti-pattern but might be a good > > gateway to accomodate migration. > > > > Another way to do this same thing might be to simply deploy a set of JVM > > containers in a cluster, which could serve as a "bigtop base image" (we > do > > this now with the bigtop tool chain, sorta), in fact, we already publish > > containers for this iirc. This would break the containerized idiom that > > is most popular (microservices), but might be an interesting staging > ground > > for people to deploy static hadoop clusters in a containerized > microsystem > > environment. > > I guess my biggest question is this: how do you see Kubernetes interact > with things like Puppet, etc? Note that even our Juju integration is done > via Puppet code today. Will this be the same for Kubernetes? > > Thanks, > Roman. >
