If only compute node (TaskTracker or NodeManager) in your instance, then decommission nodes and shutdown related EC2 instances should be fine although some finished/running tasks might need to be re-run automatically. If future, we would support gracefully decommission (tracked by YARN-914 and MAPREDUCE-5381) so that no tasks need to be rerun in this case (but need to wait a while).
Thanks, Junping ----- Original Message ----- From: "Nan Zhu" <[email protected]> To: [email protected] Sent: Wednesday, October 23, 2013 8:15:51 PM Subject: Re: dynamically resizing Hadoop cluster on AWS? Oh, I’m not running HDFS in the instances, I use S3 to save data -- Nan Zhu School of Computer Science, McGill University On Wednesday, October 23, 2013 at 11:11 PM, Nan Zhu wrote: > Hi, all > > I’m running a Hadoop cluster on AWS EC2, > > I would like to dynamically resizing the cluster so as to reduce the cost, is > there any solution to achieve this? > > E.g. I would like to cut the cluster size with a half, is it safe to just > shutdown the instances (if some tasks are just running on them, can I rely on > the speculative execution to re-run them on other nodes?) > > I cannot use EMR, since I’m running a customized version of Hadoop > > Best, > > -- > Nan Zhu > School of Computer Science, > McGill University >
