Either of the days is open for me: would love to have a demo. Perhaps it can be recorded to be used as a screencast later on? ;)
It'd be great to have as many people as possible on that, so we can quickly converge on the directions here. Adding user@ list as well... Cos On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote: > Thanks Jay, > A screencast is something we have had on our todo list; we can easily > prioritize a bigtop example as the first of a few examples we were planning > to do. > > As far as the demo, some dates we had in mind were Wednesday or Thursday > next week. > I guess the easiest way to see who is interested is a reply/+1 > with Wednesday or Thursday; we then can find exact times. > We are happy to schedule multiple demos at a few different days/times. > -Rich > > > ----- Original Message ----- > From: > [email protected] > > To: > "[email protected]" <[email protected]> > Cc: > > Sent: > Wed, 25 Feb 2015 12:32:59 -0500 > Subject: > Re: polishing off bigtop : SSH Provisioning to the cloud > > > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop > would be awesome ! > > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <[email protected]> wrote: > > > I think might have a solution (dtk.io) that we have worked on that can > > help > > solve these problems. We have been working the last four years on the > > problem of deploying complex clusters. It provides a simple DSL for > > describing a rich set of topologies (eg simple master slave topologies, HA > > variants, security variants, monitoring and centralized logging > > extensions). It provides one click deployment from the selected cluster > > topology. It also provides a simple workflow DSL to enable finer grain > > control on order of execution. > > > > We are in the process now of refining a plan to open source it > > > > Our system was built to work on top of user's existing configuration > > assets. The initial focus has been on Puppet, but we are expanding now for > > docker support and using arbitrary scripts > > > > For last two years we have been using some of the Bigtop puppet modules > for > > our big data deployments as well as in concert with modules such as kafka, > > storm, accumulo , opentsdb ,.. We are in process now of converting to use > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to > > leverage the groovy and gradle test work so one can bring up cluster stage > > by stage with smoke tests after every stage > > > > Think it would be straight forward so this can also plug on top of Bigtop > > vagrant work; we have a plugable iaas architecture; we currently support > > ec2 and "managed servers" (we have focused here on bare metal); so vagrant > > would extend reach to virtual box and its other iaas providers. > > > > I think there is potential for great synergy; best way to show would be > > through > > goto meeting demo(s) and answering any detailed questions. We can do this > > next week or the following at one or a few sessions. Will shortly send > > proposed dates/times > > - Rich > > > > > > > > ----- Original Message ----- > > From: > > [email protected] > > > > To: > > <[email protected]> > > Cc: > > > > Sent: > > Wed, 25 Feb 2015 17:24:41 +0800 > > Subject: > > Re: polishing off bigtop : SSH Provisioning to the cloud. > > > > > > Cluster deployment and management are important things to make our work > > finally adopted by users. From my point of view I think there are two > > things we can improve further: > > * user friendly cluster auto deployment > > (we have puppet recipes, but users still need to manually ship puppet > > code and configurations) > > * topology layout > > (for example, specify zookeeper to be deployed on node 3~5) > > > > BIGTOP-1702 has a great potential to make cluster deployment more user > > friendly. > > That is users don't need to deal with those trivial things like ship > bigtop > > puppet code and hiera configurations or install those prerequisites all > > over the cluster. > > > > To answer cos's question in BIGTOP-1702: > > > > although I am not sure what you mean by > > link those managed servers for further provisioning > > > > According to the readme of vagrant-managed-servers, vagrant up and destroy > > are re-interpreted as "linking//unlinking". After those servers are > > linked, theoretically, we can directly leverage our Vagrantfile > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click. > > So, this definitely will take advantage of existing puppet recipes. > > > > And here's my thought on following questions: > > > > - how that topology will be specified? > > - how it is going to be compatible with what we have right now? Not like I > > try to preserve what we have right now, as there's not much to preserve. > > We don't really have a way to desribe a cluster's topology at the > > moment anyway. > > - in case we need to describe more complex topology - how would be done? > > > > I don't think BIGTOP-1702 support topology settings. Regarding to the > > feature, I think we should support topology specification in our puppet > > recipes. We currently do not support this yet, but I believe Michael is > > heading toward this way. > > > > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > I think it is a great idea! > > > > > > And, indeed, it's just a very small step - make it user-friendly ;) > > > > > > I believe what you're proposing below is a much better way to achieve > > what > > > I've been trying by wrapping Puppet and content distribution into Gradle > > > (you > > > referred to it 'our own deployer' below). That was a bad idea, in > > > retrospect. > > > And for sure - we need the functionality of this kind: we need something > > > that > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly > without > > > additional troubles! > > > > > > I am also a big fan of getting rid of lengthy and at times cryptic pages > > > explaining the installation process. > > > > > > Now, the solution you have in mind will be helping with a typical > cluster > > > topology like we are usually setting up with head_node, and other flat > > > things > > > of this nature, right? In other words - a simple, "traditional" cluster > > > layout? If so, may I ask a few questions: > > > - how that topology will be specified? > > > - how it is going to be compatible with what we have right now? Not like > > I > > > try to preserve what we have right now, as there's not much to preserve. > > > We don't really have a way to desribe a cluster's topology at the > > > moment anyway. > > > - in case we need to describe more complex topology - how would be done? > > > > > > I also presume, that the proposed solution will take advantage of > > existing > > > Puppet recipes, right? > > > > > > Thanks in advance for your answers! > > > Cos > > > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote: > > > > Hi folks. > > > > > > > > This last year we've made a lot of progress, > > > > - we can now easily build bigtop, > > > > - and we can test a fresh build in output/ it in vagrant boxes locally > > > > > > > > Whats next? > > > > > > > > In my mind, there is only one last step, to make bigtop a consumer > > facing > > > > product: We need to make it so you can simply provision a cluster in > > the > > > > cloud. > > > > As you may recall, David (student at RPI, intern at analytics company > > > last > > > > summer), asked how to do this, and spent about three weeks on it. > > > Recently > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston University, > > and > > > > former Yahoo researchers, also asked me the same thing. > > > > > > > > Alternative? > > > > > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so > > on, > > > > which mostly are an untested, human readable version of our vagrant > > > > recipes. yikes ! > > > > > > > > How to implement a ssh cloud bigtop installation ? > > > > > > > > We could roll our own deployer, but in doing so, we would have to make > > > > semantics for: > > > > > > > > - possible need for machiene reboots > > > > - syncing local folders to remote folders > > > > - installing (and reinstalling) > > > > > > > > luckily, our buddy Vagrant already does this for us (respectively , > > with > > > > the "reload","synced.folder", and "up" options). > > > > > > > > Additionally, *** I THINK *** we can literally use the exact same > > vagrant > > > > recipes which we are already using to test --- so we will have a great > > > user > > > > experience, and really easy to reproduce bugs and test deployments. > > > > > > > > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions, > > > suggestions > > > > on the implementation or feature? I think it will be easy, but also, > it > > > > will be one of the most powerful things we add to bigtop, basically > > > > allowing users to easily use the system... maybe even too easy :) > > > > > > > > -- > > > > jay vyas > > > > > > > > > -- > jay vyas
