Re: Cluster Management
On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso wrote: > Puppet was the first thing that came to mind for us as well. In addition, > we had the same thought about auto-restarting nodes when the configuration > is changed. If a configuration on all the nodes is changed, we would want > to restart one node at a time and wait for it to rejoin before restarting > the next one. I am assuming in a case like this, you then manually perform > the restart operation for each node? > FYI, potentially relevant : https://issues.apache.org/jira/browse/CASSANDRA-2356 =Rob
Re: Cluster Management
An alternative to cssh is fabric. It's very flexible in that you can automate almost any repetitive task that you'd send to machines in a cluster, and it's written in python, meaning if you're in AWS you can mix it with boto to automate pretty much anything you want. On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso wrote: > Hi Particia, > > Thank you for the feedback. It has been helpful. > > > On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla > wrote: > >> Anthony, >> >> We use a number of tools to manage our Cassandra cluster. >> >> * Datastax OpsCenter [0] for at a glance information, and trending >> statistics. You can also run operations through here, though I prefer >> to use nodetool for any mutative operation. >> * nodetool for ad hoc status checks, and day-to-day node management. >> * puppet for setup and initialization >> >> > For example, if I want to make some changes to the configuration file >> that resides on each node, is there a tool that will propagate the change >> to each node? >> >> For this, we use puppet to manage any changes to the configurations >> (which are stored in git). We initially had Cassandra auto-restart >> when the configuration changed, but you might not want the node to >> automatically join a cluster, so we turned this off. >> > > Puppet was the first thing that came to mind for us as well. In addition, > we had the same thought about auto-restarting nodes when the configuration > is changed. If a configuration on all the nodes is changed, we would want > to restart one node at a time and wait for it to rejoin before restarting > the next one. I am assuming in a case like this, you then manually perform > the restart operation for each node? > > >> >> > Another example is if I want to have a rolling repair (nodetool repair >> -pr) and clean up running on my cluster, is there a tool that will help >> manage/configure that? >> >> Multiple commands to the cluster are sent via clusterssh [1] (cssh for >> OS X). I can easily choose which nodes to control, and run those in >> sync. For any rolling procedures, we send commands one at a time, >> though we've considered sending some of these tasks to cron. >> > > Thanks again for the tip! This is quite interesting; it may help to solve > our immediate problem for now. > > Regards, > Anthony > > >> >> Hope this helps. >> >> Cheers, >> Patricia >> >> >> [0] http://planetcassandra.org/Download/DataStaxCommunityEdition >> [1] http://sourceforge.net/projects/clusterssh/ >> > > -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: Cluster Management
Thanks Nate! We will look into this one to see if we can use it. Regards, Anthony On Tue, Aug 27, 2013 at 12:22 AM, Nate McCall wrote: > >> For example, if I want to make some changes to the configuration file >> that resides on each node, is there a tool that will propagate the change >> to each node? >> >> You may also want to take a look at Priam from the Netflix folks: > https://github.com/Netflix/Priam > > Assumes AWS (though some of this is becoming more plug-able). >
Re: Cluster Management
Hi Particia, Thank you for the feedback. It has been helpful. On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla wrote: > Anthony, > > We use a number of tools to manage our Cassandra cluster. > > * Datastax OpsCenter [0] for at a glance information, and trending > statistics. You can also run operations through here, though I prefer > to use nodetool for any mutative operation. > * nodetool for ad hoc status checks, and day-to-day node management. > * puppet for setup and initialization > > > For example, if I want to make some changes to the configuration file > that resides on each node, is there a tool that will propagate the change > to each node? > > For this, we use puppet to manage any changes to the configurations > (which are stored in git). We initially had Cassandra auto-restart > when the configuration changed, but you might not want the node to > automatically join a cluster, so we turned this off. > Puppet was the first thing that came to mind for us as well. In addition, we had the same thought about auto-restarting nodes when the configuration is changed. If a configuration on all the nodes is changed, we would want to restart one node at a time and wait for it to rejoin before restarting the next one. I am assuming in a case like this, you then manually perform the restart operation for each node? > > > Another example is if I want to have a rolling repair (nodetool repair > -pr) and clean up running on my cluster, is there a tool that will help > manage/configure that? > > Multiple commands to the cluster are sent via clusterssh [1] (cssh for > OS X). I can easily choose which nodes to control, and run those in > sync. For any rolling procedures, we send commands one at a time, > though we've considered sending some of these tasks to cron. > Thanks again for the tip! This is quite interesting; it may help to solve our immediate problem for now. Regards, Anthony > > Hope this helps. > > Cheers, > Patricia > > > [0] http://planetcassandra.org/Download/DataStaxCommunityEdition > [1] http://sourceforge.net/projects/clusterssh/ >
Re: Cluster Management
> > > For example, if I want to make some changes to the configuration file that > resides on each node, is there a tool that will propagate the change to > each node? > > You may also want to take a look at Priam from the Netflix folks: https://github.com/Netflix/Priam Assumes AWS (though some of this is becoming more plug-able).
Re: Cluster Management
Anthony, We use a number of tools to manage our Cassandra cluster. * Datastax OpsCenter [0] for at a glance information, and trending statistics. You can also run operations through here, though I prefer to use nodetool for any mutative operation. * nodetool for ad hoc status checks, and day-to-day node management. * puppet for setup and initialization > For example, if I want to make some changes to the configuration file that > resides on each node, is there a tool that will propagate the change to each > node? For this, we use puppet to manage any changes to the configurations (which are stored in git). We initially had Cassandra auto-restart when the configuration changed, but you might not want the node to automatically join a cluster, so we turned this off. > Another example is if I want to have a rolling repair (nodetool repair -pr) > and clean up running on my cluster, is there a tool that will help > manage/configure that? Multiple commands to the cluster are sent via clusterssh [1] (cssh for OS X). I can easily choose which nodes to control, and run those in sync. For any rolling procedures, we send commands one at a time, though we've considered sending some of these tasks to cron. Hope this helps. Cheers, Patricia [0] http://planetcassandra.org/Download/DataStaxCommunityEdition [1] http://sourceforge.net/projects/clusterssh/
