Count me in. Thanks! Cos
On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote: > We set up a goto meeting for 10:00AM PST on Thursday March 5th > > *Bigtop - DevOps Toolkit (DTK) Demo* > Thu, Mar 5, 10:00 AM Pacific Standard Time > > > - Please join my meeting from your computer, tablet or smartphone. > https://global.gotomeeting.com/join/655931173 > - You can also dial in using your phone. > United States (Long distance): +1 (571) 317-3122 > *Access Code:* 655-931-173 > More phone numbers: > https://global.gotomeeting.com/655931173/numbersdisplay.html > > If interested, but this day or time does not work, please send a note and > we can set up additional day/time(s) too. > > We will have a few high level slides and then have a demo that shows how we > currently use Bigtop and how this can be naturally extended .to provide a > "one click" solution to deploy a rich variety of bigtop stacks with > different mixtures of services and different topologies that plugs on top > of the existing bigtop deployment work > > As far as a screencast, we wanted our first one to be a short focused > screencast; so that will be a follow up that we will put together shortly > - Rich > > > > > ----- Original Message ----- > From: > [email protected] > > To: > <[email protected]> > Cc: > <[email protected]> > Sent: > Sat, 28 Feb 2015 01:31:10 +0000 > Subject: > Re: polishing off bigtop : SSH Provisioning to the cloud > > > Either of the days is open for me: would love to have a demo. Perhaps it can > be recorded to be used as a screencast later on? ;) > > It'd be great to have as many people as possible on that, so we can quickly > converge on the directions here. Adding user@ list as well... > > Cos > > On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote: > > Thanks Jay, > > A screencast is something we have had on our todo list; we can easily > > prioritize a bigtop example as the first of a few examples we were > planning > > to do. > > > > As far as the demo, some dates we had in mind were Wednesday or Thursday > > next week. > > I guess the easiest way to see who is interested is a reply/+1 > > with Wednesday or Thursday; we then can find exact times. > > We are happy to schedule multiple demos at a few different days/times. > > -Rich > > > > > > ----- Original Message ----- > > From: > > [email protected] > > > > To: > > "[email protected]" <[email protected]> > > Cc: > > > > Sent: > > Wed, 25 Feb 2015 12:32:59 -0500 > > Subject: > > Re: polishing off bigtop : SSH Provisioning to the cloud > > > > > > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop > > would be awesome ! > > > > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <[email protected]> > wrote: > > > > > I think might have a solution (dtk.io) that we have worked on that can > > > help > > > solve these problems. We have been working the last four years on the > > > problem of deploying complex clusters. It provides a simple DSL for > > > describing a rich set of topologies (eg simple master slave topologies, > HA > > > variants, security variants, monitoring and centralized logging > > > extensions). It provides one click deployment from the selected cluster > > > topology. It also provides a simple workflow DSL to enable finer grain > > > control on order of execution. > > > > > > We are in the process now of refining a plan to open source it > > > > > > Our system was built to work on top of user's existing configuration > > > assets. The initial focus has been on Puppet, but we are expanding now > for > > > docker support and using arbitrary scripts > > > > > > For last two years we have been using some of the Bigtop puppet modules > > for > > > our big data deployments as well as in concert with modules such as > kafka, > > > storm, accumulo , opentsdb ,.. We are in process now of converting to > use > > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to > > > leverage the groovy and gradle test work so one can bring up cluster > stage > > > by stage with smoke tests after every stage > > > > > > Think it would be straight forward so this can also plug on top of > Bigtop > > > vagrant work; we have a plugable iaas architecture; we currently support > > > ec2 and "managed servers" (we have focused here on bare metal); so > vagrant > > > would extend reach to virtual box and its other iaas providers. > > > > > > I think there is potential for great synergy; best way to show would be > > > through > > > goto meeting demo(s) and answering any detailed questions. We can do > this > > > next week or the following at one or a few sessions. Will shortly send > > > proposed dates/times > > > - Rich > > > > > > > > > > > > ----- Original Message ----- > > > From: > > > [email protected] > > > > > > To: > > > <[email protected]> > > > Cc: > > > > > > Sent: > > > Wed, 25 Feb 2015 17:24:41 +0800 > > > Subject: > > > Re: polishing off bigtop : SSH Provisioning to the cloud. > > > > > > > > > Cluster deployment and management are important things to make our work > > > finally adopted by users. From my point of view I think there are two > > > things we can improve further: > > > * user friendly cluster auto deployment > > > (we have puppet recipes, but users still need to manually ship puppet > > > code and configurations) > > > * topology layout > > > (for example, specify zookeeper to be deployed on node 3~5) > > > > > > BIGTOP-1702 has a great potential to make cluster deployment more user > > > friendly. > > > That is users don't need to deal with those trivial things like ship > > bigtop > > > puppet code and hiera configurations or install those prerequisites all > > > over the cluster. > > > > > > To answer cos's question in BIGTOP-1702: > > > > > > although I am not sure what you mean by > > > link those managed servers for further provisioning > > > > > > According to the readme of vagrant-managed-servers, vagrant up and > destroy > > > are re-interpreted as "linking//unlinking". After those servers are > > > linked, theoretically, we can directly leverage our Vagrantfile > > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click. > > > So, this definitely will take advantage of existing puppet recipes. > > > > > > And here's my thought on following questions: > > > > > > - how that topology will be specified? > > > - how it is going to be compatible with what we have right now? Not > like I > > > try to preserve what we have right now, as there's not much to preserve. > > > We don't really have a way to desribe a cluster's topology at the > > > moment anyway. > > > - in case we need to describe more complex topology - how would be done? > > > > > > I don't think BIGTOP-1702 support topology settings. Regarding to the > > > feature, I think we should support topology specification in our puppet > > > recipes. We currently do not support this yet, but I believe Michael is > > > heading toward this way. > > > > > > > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > I think it is a great idea! > > > > > > > > And, indeed, it's just a very small step - make it user-friendly ;) > > > > > > > > I believe what you're proposing below is a much better way to achieve > > > what > > > > I've been trying by wrapping Puppet and content distribution into > Gradle > > > > (you > > > > referred to it 'our own deployer' below). That was a bad idea, in > > > > retrospect. > > > > And for sure - we need the functionality of this kind: we need > something > > > > that > > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly > > without > > > > additional troubles! > > > > > > > > I am also a big fan of getting rid of lengthy and at times cryptic > pages > > > > explaining the installation process. > > > > > > > > Now, the solution you have in mind will be helping with a typical > > cluster > > > > topology like we are usually setting up with head_node, and other flat > > > > things > > > > of this nature, right? In other words - a simple, "traditional" > cluster > > > > layout? If so, may I ask a few questions: > > > > - how that topology will be specified? > > > > - how it is going to be compatible with what we have right now? Not > like > > > I > > > > try to preserve what we have right now, as there's not much to > preserve. > > > > We don't really have a way to desribe a cluster's topology at the > > > > moment anyway. > > > > - in case we need to describe more complex topology - how would be > done? > > > > > > > > I also presume, that the proposed solution will take advantage of > > > existing > > > > Puppet recipes, right? > > > > > > > > Thanks in advance for your answers! > > > > Cos > > > > > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote: > > > > > Hi folks. > > > > > > > > > > This last year we've made a lot of progress, > > > > > - we can now easily build bigtop, > > > > > - and we can test a fresh build in output/ it in vagrant boxes > locally > > > > > > > > > > Whats next? > > > > > > > > > > In my mind, there is only one last step, to make bigtop a consumer > > > facing > > > > > product: We need to make it so you can simply provision a cluster in > > > the > > > > > cloud. > > > > > As you may recall, David (student at RPI, intern at analytics > company > > > > last > > > > > summer), asked how to do this, and spent about three weeks on it. > > > > Recently > > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston > University, > > > and > > > > > former Yahoo researchers, also asked me the same thing. > > > > > > > > > > Alternative? > > > > > > > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so > > > on, > > > > > which mostly are an untested, human readable version of our vagrant > > > > > recipes. yikes ! > > > > > > > > > > How to implement a ssh cloud bigtop installation ? > > > > > > > > > > We could roll our own deployer, but in doing so, we would have to > make > > > > > semantics for: > > > > > > > > > > - possible need for machiene reboots > > > > > - syncing local folders to remote folders > > > > > - installing (and reinstalling) > > > > > > > > > > luckily, our buddy Vagrant already does this for us (respectively , > > > with > > > > > the "reload","synced.folder", and "up" options). > > > > > > > > > > Additionally, *** I THINK *** we can literally use the exact same > > > vagrant > > > > > recipes which we are already using to test --- so we will have a > great > > > > user > > > > > experience, and really easy to reproduce bugs and test deployments. > > > > > > > > > > > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions, > > > > suggestions > > > > > on the implementation or feature? I think it will be easy, but also, > > it > > > > > will be one of the most powerful things we add to bigtop, basically > > > > > allowing users to easily use the system... maybe even too easy :) > > > > > > > > > > -- > > > > > jay vyas > > > > > > > > > > > > > > > -- > > jay vyas
