Evans We are not set up to record tomorrow's session, but can record something shortly. Also happy to do a demo that meets your time zone; so if interested please suggest some times that work and we can accommodate - Rich
----- Original Message ----- From: [email protected] To: Cc: Sent:Thu, 5 Mar 2015 14:21:35 +0800 Subject:Re: polishing off bigtop : SSH Provisioning to the cloud Hi Richard, Thanks for setting up this. I can't join the meeting because of time zone difference. I think it would be great to have this recorded:) Evans 2015-03-05 13:15 GMT+08:00 Konstantin Boudnik : > Guys, just want to remind everyone that we have the DTK demo tomorrow 10 > PST > - should be great! > > Cos > > On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote: > > We set up a goto meeting for 10:00AM PST on Thursday March 5th > > > > *Bigtop - DevOps Toolkit (DTK) Demo* > > Thu, Mar 5, 10:00 AM Pacific Standard Time > > > > > > - Please join my meeting from your computer, tablet or smartphone. > > https://global.gotomeeting.com/join/655931173 > > - You can also dial in using your phone. > > United States (Long distance): +1 (571) 317-3122 > > *Access Code:* 655-931-173 > > More phone numbers: > > https://global.gotomeeting.com/655931173/numbersdisplay.html > > > > If interested, but this day or time does not work, please send a note and > > we can set up additional day/time(s) too. > > > > We will have a few high level slides and then have a demo that shows how > we > > currently use Bigtop and how this can be naturally extended .to provide a > > "one click" solution to deploy a rich variety of bigtop stacks with > > different mixtures of services and different topologies that plugs on top > > of the existing bigtop deployment work > > > > As far as a screencast, we wanted our first one to be a short focused > > screencast; so that will be a follow up that we will put together shortly > > - Rich > > > > > > > > > > ----- Original Message ----- > > From: > > [email protected] > > > > To: > > > > Cc: > > > > Sent: > > Sat, 28 Feb 2015 01:31:10 +0000 > > Subject: > > Re: polishing off bigtop : SSH Provisioning to the cloud > > > > > > Either of the days is open for me: would love to have a demo. Perhaps it > can > > be recorded to be used as a screencast later on? ;) > > > > It'd be great to have as many people as possible on that, so we can > quickly > > converge on the directions here. Adding user@ list as well... > > > > Cos > > > > On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote: > > > Thanks Jay, > > > A screencast is something we have had on our todo list; we can easily > > > prioritize a bigtop example as the first of a few examples we were > > planning > > > to do. > > > > > > As far as the demo, some dates we had in mind were Wednesday or > Thursday > > > next week. > > > I guess the easiest way to see who is interested is a reply/+1 > > > with Wednesday or Thursday; we then can find exact times. > > > We are happy to schedule multiple demos at a few different days/times. > > > -Rich > > > > > > > > > ----- Original Message ----- > > > From: > > > [email protected] > > > > > > To: > > > "[email protected]" > > > Cc: > > > > > > Sent: > > > Wed, 25 Feb 2015 12:32:59 -0500 > > > Subject: > > > Re: polishing off bigtop : SSH Provisioning to the cloud > > > > > > > > > Hi richard, also maybe even a youtube video of Devops TK deploying > BigTop > > > would be awesome ! > > > > > > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin > > wrote: > > > > > > > I think might have a solution (dtk.io) that we have worked on that > can > > > > help > > > > solve these problems. We have been working the last four years on the > > > > problem of deploying complex clusters. It provides a simple DSL for > > > > describing a rich set of topologies (eg simple master slave > topologies, > > HA > > > > variants, security variants, monitoring and centralized logging > > > > extensions). It provides one click deployment from the selected > cluster > > > > topology. It also provides a simple workflow DSL to enable finer > grain > > > > control on order of execution. > > > > > > > > We are in the process now of refining a plan to open source it > > > > > > > > Our system was built to work on top of user's existing configuration > > > > assets. The initial focus has been on Puppet, but we are expanding > now > > for > > > > docker support and using arbitrary scripts > > > > > > > > For last two years we have been using some of the Bigtop puppet > modules > > > for > > > > our big data deployments as well as in concert with modules such as > > kafka, > > > > storm, accumulo , opentsdb ,. We are in process now of converting to > > use > > > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability > to > > > > leverage the groovy and gradle test work so one can bring up cluster > > stage > > > > by stage with smoke tests after every stage > > > > > > > > Think it would be straight forward so this can also plug on top of > > Bigtop > > > > vagrant work; we have a plugable iaas architecture; we currently > support > > > > ec2 and "managed servers" (we have focused here on bare metal); so > > vagrant > > > > would extend reach to virtual box and its other iaas providers. > > > > > > > > I think there is potential for great synergy; best way to show would > be > > > > through > > > > goto meeting demo(s) and answering any detailed questions. We can do > > this > > > > next week or the following at one or a few sessions. Will shortly > send > > > > proposed dates/times > > > > - Rich > > > > > > > > > > > > > > > > ----- Original Message ----- > > > > From: > > > > [email protected] > > > > > > > > To: > > > > > > > > Cc: > > > > > > > > Sent: > > > > Wed, 25 Feb 2015 17:24:41 +0800 > > > > Subject: > > > > Re: polishing off bigtop : SSH Provisioning to the cloud. > > > > > > > > > > > > Cluster deployment and management are important things to make our > work > > > > finally adopted by users. From my point of view I think there are two > > > > things we can improve further: > > > > * user friendly cluster auto deployment > > > > (we have puppet recipes, but users still need to manually ship puppet > > > > code and configurations) > > > > * topology layout > > > > (for example, specify zookeeper to be deployed on node 3~5) > > > > > > > > BIGTOP-1702 has a great potential to make cluster deployment more > user > > > > friendly. > > > > That is users don't need to deal with those trivial things like ship > > > bigtop > > > > puppet code and hiera configurations or install those prerequisites > all > > > > over the cluster. > > > > > > > > To answer cos's question in BIGTOP-1702: > > > > > > > > although I am not sure what you mean by > > > > link those managed servers for further provisioning > > > > > > > > According to the readme of vagrant-managed-servers, vagrant up and > > destroy > > > > are re-interpreted as "linking//unlinking". After those servers are > > > > linked, theoretically, we can directly leverage our Vagrantfile > > > > in "vagrant-puppet-vm" to do the whole cluster deployment by > one-click. > > > > So, this definitely will take advantage of existing puppet recipes. > > > > > > > > And here's my thought on following questions: > > > > > > > > - how that topology will be specified? > > > > - how it is going to be compatible with what we have right now? Not > > like I > > > > try to preserve what we have right now, as there's not much to > preserve. > > > > We don't really have a way to desribe a cluster's topology at the > > > > moment anyway. > > > > - in case we need to describe more complex topology - how would be > done? > > > > > > > > I don't think BIGTOP-1702 support topology settings. Regarding to the > > > > feature, I think we should support topology specification in our > puppet > > > > recipes. We currently do not support this yet, but I believe Michael > is > > > > heading toward this way. > > > > > > > > > > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik : > > > > > > > > > I think it is a great idea! > > > > > > > > > > And, indeed, it's just a very small step - make it user-friendly ;) > > > > > > > > > > I believe what you're proposing below is a much better way to > achieve > > > > what > > > > > I've been trying by wrapping Puppet and content distribution into > > Gradle > > > > > (you > > > > > referred to it 'our own deployer' below) That was a bad idea, in > > > > > retrospect. > > > > > And for sure - we need the functionality of this kind: we need > > something > > > > > that > > > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly > > > without > > > > > additional troubles! > > > > > > > > > > I am also a big fan of getting rid of lengthy and at times cryptic > > pages > > > > > explaining the installation process. > > > > > > > > > > Now, the solution you have in mind will be helping with a typical > > > cluster > > > > > topology like we are usually setting up with head_node, and other > flat > > > > > things > > > > > of this nature, right? In other words - a simple, "traditional" > > cluster > > > > > layout? If so, may I ask a few questions: > > > > > - how that topology will be specified? > > > > > - how it is going to be compatible with what we have right now? Not > > like > > > > I > > > > > try to preserve what we have right now, as there's not much to > > preserve. > > > > > We don't really have a way to desribe a cluster's topology at the > > > > > moment anyway. > > > > > - in case we need to describe more complex topology - how would be > > done? > > > > > > > > > > I also presume, that the proposed solution will take advantage of > > > > existing > > > > > Puppet recipes, right? > > > > > > > > > > Thanks in advance for your answers! > > > > > Cos > > > > > > > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote: > > > > > > Hi folks. > > > > > > > > > > > > This last year we've made a lot of progress, > > > > > > - we can now easily build bigtop, > > > > > > - and we can test a fresh build in output/ it in vagrant boxes > > locally > > > > > > > > > > > > Whats next? > > > > > > > > > > > > In my mind, there is only one last step, to make bigtop a > consumer > > > > facing > > > > > > product: We need to make it so you can simply provision a > cluster in > > > > the > > > > > > cloud. > > > > > > As you may recall, David (student at RPI, intern at analytics > > company > > > > > last > > > > > > summer), asked how to do this, and spent about three weeks on it. > > > > > Recently > > > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston > > University, > > > > and > > > > > > former Yahoo researchers, also asked me the same thing. > > > > > > > > > > > > Alternative? > > > > > > > > > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, > and so > > > > on, > > > > > > which mostly are an untested, human readable version of our > vagrant > > > > > > recipes. yikes ! > > > > > > > > > > > > How to implement a ssh cloud bigtop installation ? > > > > > > > > > > > > We could roll our own deployer, but in doing so, we would have to > > make > > > > > > semantics for: > > > > > > > > > > > > - possible need for machiene reboots > > > > > > - syncing local folders to remote folders > > > > > > - installing (and reinstalling) > > > > > > > > > > > > luckily, our buddy Vagrant already does this for us > (respectively , > > > > with > > > > > > the "reload","synced.folder", and "up" options). > > > > > > > > > > > > Additionally, *** I THINK *** we can literally use the exact same > > > > vagrant > > > > > > recipes which we are already using to test --- so we will have a > > great > > > > > user > > > > > > experience, and really easy to reproduce bugs and test > deployments. > > > > > > > > > > > > > > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions, > > > > > suggestions > > > > > > on the implementation or feature? I think it will be easy, but > also, > > > it > > > > > > will be one of the most powerful things we add to bigtop, > basically > > > > > > allowing users to easily use the system... maybe even too easy :) > > > > > > > > > > > > -- > > > > > > jay vyas > > > > > > > > > > > > > > > > > > > > > -- > > > jay vyas >
