Regarding to the puppetmaster, I'd like to raise a question that does most of the company using puppet in this way? I have experience using puppet master for cluster management and deployment. But there are troubles I ran into and I think it would be better to switch to masterless puppet. When I searched around on net, I found that actually there are many companies already did the switch. A typical solution for using masterless puppet is to use version control(git) with puppet apply. (though I'm not sure whether there is a better way)
To increase the Bigtop adoption by improving our puppet stuff, I think there are some items can be put on for discussion: * Journal HA for Namenodes * find-grained configuration on roles (some machines run datanode+zookeeper+journalnode, some machines only run nodemanager) However, this is only my personal thoughts. Would love to hear yours. Getting back to the topic, I think the gradle thing can better demonstrate the usage of bigtop puppet, though some of the ops people might not want the extra dependency:) 2014-12-20 12:17 GMT+08:00 Konstantin Boudnik <[email protected]>: > Thanks for kind words, man - I am glad to see that Gradle took off! > > Let me clarify something - sorry for not being clear right away. The > application of the wrapper isn't to manage LIVE clusters or machines. Also, > Puppet recipes aren't going away neither. All I had in mind is to provide > an > easy wrapper, so that > > a) don't bother with copying around deployment script when you need to > quickly > get a dev. cluster up and running > b) one doesn't need to type the long command wrapped in pdsh or otherwise > > It might be or might not be a good idea and that's why I wanted to get it > out > there for everyone to bash on ;) > > Cos > > On Fri, Dec 19, 2014 at 09:25PM, jay vyas wrote: > > Hi cos! I like the goal, but let me pervert your idea a little bit :) > .... > > how does this sound ? > > > > *** Definetely *** > > > > lets keep going w/ gradle for all build/test/deploy stuff. Its great for > > that ! Full court press to remove maven deps, gradle is wayyyy cleaner > and > > definetely is the future trend. the hard work you've done in integrating > > the whole build to use it is really making bigtop a fun to use packaging > > system, and actually has made me more able to contribute to it b/c its > > easier to debug the build now for guys like me that dont know much about > > make. > > > > *** BUT *** > > > > although I LOVE the idea of adding a CONTROLLER to the live machines, ... > > adding it in gradle? very few devops folks will really want to use gradle > > for that kind of task.... its not the idiom. > > > > One of the HUGE selling points of bigtop is that it has puppet recipes, > > rather than just one of commands/tasks for managing the cluster - the ops > > community can read and understand those easily. > > > > *** SO : PUPPET MASTER .... AN ALTERNATIVE ? *** > > > > I think, if we want to manage LIVE machines, then can i suggest a real > > puppet master implementation? If we go that route, you get all the > testing > > for it also, for free, w/ our existing vagrant recipes.... and its in > line > > w/ what the devops community is doing, and so it will help drive bigtop > > adoption. > > > > > > On Fri, Dec 19, 2014 at 8:19 PM, Konstantin Boudnik <[email protected]> > wrote: > > > > > > Guys, > > > > > > I've been doing some thinking lately (yes, I know I shouldn't - I've > heard > > > this from good friends before, but can't help it) and it makes more and > > > more > > > sense to me to wrap more build/deploy functionality into our Gradle > build. > > > > > > The low-hanging fruits I have in mind right now is to provide a > wrapper for > > > {{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a > few > > > things like: > > > - copy deployment recipes to all specified hosts, including site.csv > file > > > - start puppet apply on all hosts in parallel using gradle ssh plugin > > > > > > This, IMO, will provide us with a very easy and transparent way of > quickly > > > deploying cluster bits as one works on something, without switching > out of > > > the > > > convenient build environment. Sorta, change-deploy-rinse-and-repeat > loop. > > > > > > The same gradle task can be called, if desired from Jenkins or else. > > > > > > Do you think it worth it? Thoughts? > > > -- > > > Take care, > > > Cos > > > > > > > > > > -- > > jay vyas >
