Hi richard, also maybe even a youtube video of Devops TK deploying BigTop would be awesome !
On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <[email protected]> wrote: > I think might have a solution (dtk.io) that we have worked on that can > help > solve these problems. We have been working the last four years on the > problem of deploying complex clusters. It provides a simple DSL for > describing a rich set of topologies (eg simple master slave topologies, HA > variants, security variants, monitoring and centralized logging > extensions). It provides one click deployment from the selected cluster > topology. It also provides a simple workflow DSL to enable finer grain > control on order of execution. > > We are in the process now of refining a plan to open source it > > Our system was built to work on top of user's existing configuration > assets. The initial focus has been on Puppet, but we are expanding now for > docker support and using arbitrary scripts > > For last two years we have been using some of the Bigtop puppet modules for > our big data deployments as well as in concert with modules such as kafka, > storm, accumulo , opentsdb ,.. We are in process now of converting to use > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to > leverage the groovy and gradle test work so one can bring up cluster stage > by stage with smoke tests after every stage > > Think it would be straight forward so this can also plug on top of Bigtop > vagrant work; we have a plugable iaas architecture; we currently support > ec2 and "managed servers" (we have focused here on bare metal); so vagrant > would extend reach to virtual box and its other iaas providers. > > I think there is potential for great synergy; best way to show would be > through > goto meeting demo(s) and answering any detailed questions. We can do this > next week or the following at one or a few sessions. Will shortly send > proposed dates/times > - Rich > > > > ----- Original Message ----- > From: > [email protected] > > To: > <[email protected]> > Cc: > > Sent: > Wed, 25 Feb 2015 17:24:41 +0800 > Subject: > Re: polishing off bigtop : SSH Provisioning to the cloud. > > > Cluster deployment and management are important things to make our work > finally adopted by users. From my point of view I think there are two > things we can improve further: > * user friendly cluster auto deployment > (we have puppet recipes, but users still need to manually ship puppet > code and configurations) > * topology layout > (for example, specify zookeeper to be deployed on node 3~5) > > BIGTOP-1702 has a great potential to make cluster deployment more user > friendly. > That is users don't need to deal with those trivial things like ship bigtop > puppet code and hiera configurations or install those prerequisites all > over the cluster. > > To answer cos's question in BIGTOP-1702: > > although I am not sure what you mean by > link those managed servers for further provisioning > > According to the readme of vagrant-managed-servers, vagrant up and destroy > are re-interpreted as "linking//unlinking". After those servers are > linked, theoretically, we can directly leverage our Vagrantfile > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click. > So, this definitely will take advantage of existing puppet recipes. > > And here's my thought on following questions: > > - how that topology will be specified? > - how it is going to be compatible with what we have right now? Not like I > try to preserve what we have right now, as there's not much to preserve. > We don't really have a way to desribe a cluster's topology at the > moment anyway. > - in case we need to describe more complex topology - how would be done? > > I don't think BIGTOP-1702 support topology settings. Regarding to the > feature, I think we should support topology specification in our puppet > recipes. We currently do not support this yet, but I believe Michael is > heading toward this way. > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > I think it is a great idea! > > > > And, indeed, it's just a very small step - make it user-friendly ;) > > > > I believe what you're proposing below is a much better way to achieve > what > > I've been trying by wrapping Puppet and content distribution into Gradle > > (you > > referred to it 'our own deployer' below). That was a bad idea, in > > retrospect. > > And for sure - we need the functionality of this kind: we need something > > that > > let anyone to deploy a cluster to a pre-provisioned nodes quickly without > > additional troubles! > > > > I am also a big fan of getting rid of lengthy and at times cryptic pages > > explaining the installation process. > > > > Now, the solution you have in mind will be helping with a typical cluster > > topology like we are usually setting up with head_node, and other flat > > things > > of this nature, right? In other words - a simple, "traditional" cluster > > layout? If so, may I ask a few questions: > > - how that topology will be specified? > > - how it is going to be compatible with what we have right now? Not like > I > > try to preserve what we have right now, as there's not much to preserve. > > We don't really have a way to desribe a cluster's topology at the > > moment anyway. > > - in case we need to describe more complex topology - how would be done? > > > > I also presume, that the proposed solution will take advantage of > existing > > Puppet recipes, right? > > > > Thanks in advance for your answers! > > Cos > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote: > > > Hi folks. > > > > > > This last year we've made a lot of progress, > > > - we can now easily build bigtop, > > > - and we can test a fresh build in output/ it in vagrant boxes locally > > > > > > Whats next? > > > > > > In my mind, there is only one last step, to make bigtop a consumer > facing > > > product: We need to make it so you can simply provision a cluster in > the > > > cloud. > > > As you may recall, David (student at RPI, intern at analytics company > > last > > > summer), asked how to do this, and spent about three weeks on it. > > Recently > > > Ata Turk, working on the Massachusets Open Cloud @ Boston University, > and > > > former Yahoo researchers, also asked me the same thing. > > > > > > Alternative? > > > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so > on, > > > which mostly are an untested, human readable version of our vagrant > > > recipes. yikes ! > > > > > > How to implement a ssh cloud bigtop installation ? > > > > > > We could roll our own deployer, but in doing so, we would have to make > > > semantics for: > > > > > > - possible need for machiene reboots > > > - syncing local folders to remote folders > > > - installing (and reinstalling) > > > > > > luckily, our buddy Vagrant already does this for us (respectively , > with > > > the "reload","synced.folder", and "up" options). > > > > > > Additionally, *** I THINK *** we can literally use the exact same > vagrant > > > recipes which we are already using to test --- so we will have a great > > user > > > experience, and really easy to reproduce bugs and test deployments. > > > > > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions, > > suggestions > > > on the implementation or feature? I think it will be easy, but also, it > > > will be one of the most powerful things we add to bigtop, basically > > > allowing users to easily use the system... maybe even too easy :) > > > > > > -- > > > jay vyas > > > -- jay vyas
