Some answers inline: - the metadata files are presumably getting executed by some sort of external engine doing actual orchestration work, right? These files are, please forgive me for the lack of the better word or analogy, carry the function of topology map, and the actual installation/configuration of the nodes is still done by the good ol' Puppet. Is this correct?
Correct, meta file is for external usage, like a vagrant file which sits in bigtop dir, but is only used externally by vagrant to deploy a node. Difference is this instead of defining a single node/vm/docker-image, etc, it’s typically defining a distributed topology/cluster, and the workflow system would handle node provisioning and installing the bigtop packages. In regards to puppet question, right now there are two metadata descriptions, one that describes topologies called an assembly, the other describes "components" which acts as a interface def in bigtops case to the puppet modules so the system can load and execute the modules to handle the config and service start part of it. - with these files in place, how one can benefit from them? Is there a service that allow to plug them in? Or an orchestration engine, that could be ran on my laptop (or else)? Basically, I am trying to figure out what other pieces will be there and how they all get together. I think in initial implementation the benefit would mostly be for people deploying distributed clusters in environments like aws as the workflow system can handle automating the node deployments, networking, and software install/config. Instead of having just a single monolithic docker image, or equivalent, can have something like "spark+hdfs+zeppelin cluster" with appropriate master/worker design. Right now the workflow system is bundled as starter uber docker all-in-one container, kind of like the bigtop build image. For local laptop/dev usage, will take little thinking/work, but we can probably look at making a "localhost docker provider" that would be analogous to aws provider that could spin up and execute the workflow all on localhost using docker for node deployment vs aws. -----Original Message----- From: Konstantin Boudnik [mailto:[email protected]] Sent: Wednesday, April 6, 2016 7:57 AM To: [email protected]; [email protected] Subject: Re: adding new deployment metadata files to bigtop Hi Nate. Thanks for the decision to share the work you've been doing and help this community to advance. I want to clarify a couple of questions here: - the metadata files are presumably getting executed by some sort of external engine doing actual orchestration work, right? These files are, please forgive me for the lack of the better word or analogy, carry the function of topology map, and the actual installation/configuration of the nodes is still done by the good ol' Puppet. Is this correct? - with these files in place, how one can benefit from them? Is there a service that allow to plug them in? Or an orchestration engine, that could be ran on my laptop (or else)? Basically, I am trying to figure out what other pieces will be there and how they all get together. In general, I am very fond of having an orchestration in Bigtop, done one way or another. Thanks! Cos On Fri, Apr 01, 2016 at 05:31PM, [email protected] wrote: > Hello bigtoppers, > > Our org has been leveraging bigtop with our consulting work the past > couple years, primarily focused on deploying/managing clustered > services using the bigtop puppet modules. We typically will pull > periodic updates and merge in latest bigtop puppet changes with our > version. Recently we have had couple customer and community inquires > interested in using bigtop to start to play with various builds and > component versions, having us integrate our deployment stuff. We are > initially going to maintain fork of bigtop to get the ball rolling on > a couple projects, but are interested in adding some base set of our > meta files to bigtop so people don't have to come to us for our fork, instead > just maintain and use vanilla apache bigtop. > > The files we would like to contribute and maintain could be considered > akin to a vagrant or docker compose file, but instead of describing > single node/image, it describes a distributed service/cluster that can > run any number of services, leveraging the puppet modules for configuration. > > Here is partial reference description for a spark service: > > https://gist.github.com/kaiyzen/8041c0a0d6b717b0836a6e424bb70712 > > We are going to start prototyping in the next week or two, looking at > a file or two in the root folder, then probably unique folder in the > root that can house various service meta descriptions so people can > just use off the shelf without having to know details or write their > own by default. In general the files are just text/yaml files, and > will have no disruption to existing gradle/tooling. > > In meantime until we have something tangible, wanted to start the > conversation for community to provide any questions/feedback/etc about > approach of including a base set of service meta files
