What if we just executed the shutdown only if a running master is found on the host running the stop script and it's the only master in the cluster?
On Thu, Mar 3, 2011 at 4:48 PM, Igor Ranitovic <[email protected]> wrote: > What about adding a simple message box using bash's whiptail? > > For example: > > rs=$(cat ${HBASE_CONF_DIR}/regionservers | xargs) > > if ( whiptail --yesno "Do you want to shutdown the cluster with the > following regionserver $rs\n[y/n]" 10 40 ) > then > # proceed with the shutdown > else > # exit > fi > > > On 03/02/2011 05:23 PM, Bill Graham wrote: >> >> Hi, >> >> We had a troubling experience today that I wanted to share. Our dev >> cluster got completely shut down by a developer by mistake, without >> said developer even realizing it. Here's how... >> >> We have multiple sets of HBase configs checked into SVN that >> developers can checkout and point their HBASE_CONF_DIR to to easily >> change from developing in local mode or testing against our >> distributed dev cluster. >> >> In local mode someone might do something like this: >> >> bin/start-hbase.sh >> bin/hbase shell >> >> ... do some work ... >> >> bin/stop-hbase.sh >> >> The problem arose when a developer accidentally tried to do this with >> their HBASE_CONF_DIR pointing to our dev cluster configs. When this >> happens, the first command will add another master to the cluster and >> the last command will shut down the entire cluster. I assume this >> happens via Zookeeper somehow, since we don't have ssh keys to >> remotely start/stop as the user running the processes. >> >> So the question is, is this a bug or a feature? If it's a feature it >> seems like an incredibly dangerous one. Once our live cluster is >> running, those configs will also be needed on the client so really bad >> things could happen by mistake. >> >> thanks, >> Bill >> > >
