Hi Rohith, Is there something like shutdown hook for containers? Can you please also tell me how to use that?
Thanks, Kishore On Wed, Apr 9, 2014 at 8:34 AM, Rohith Sharma K S <rohithsharm...@huawei.com > wrote: > For local container clean up, can be cleaned at ShutDownHook. !!?? > > > > Thanks & Regards > > Rohith Sharma K S > > > > *From:* Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com] > *Sent:* 08 April 2014 20:01 > *To:* user@hadoop.apache.org > *Subject:* Re: Cleanup activity on YARN containers > > > > Hi Rohith, > > > > Thanks for the reply. > > > > Mine is a YARN application. I have some files that are local to where > the containers run on, and I want to clean them up at the end of the > container execution. So, I want to do this cleanup on the same node my > container ran on. With what you are suggesting, I can't delete the files > local to the container. > > > > Is there any other way? > > > > Thanks, > > Kishore > > > > On Tue, Apr 8, 2014 at 8:55 AM, Rohith Sharma K S < > rohithsharm...@huawei.com> wrote: > > Hi Kishore, > > > > Is jobs are submitted through MapReduce or Is it Yarn Application? > > > > 1. For MapReduce Framwork, framework itself provides facility to > clean up per task level. > > Is there any callback kind of facility, in which I can write > some code to be executed on my container at the end of my application or *at > the end of that particular container execution?* > > >>> You can override setup() and cleanup() for doing initialization and > cleanup of your task. This facility is provided by MapReduce framework. > > > > The call flow of task execution is > > The framework first calls > setup(org.apache.hadoop.mapreduce.Mapper.Context), followed by map(Object, > Object, Context) / reduce(Object, Iterable, Context) for each key/value > pair. Finally cleanup(Context) is called. > > > > Note : In clean up, do not hold container for more than > "mapreduce.task.timeout". Because, once map/reduce is completed, progress > will not be sent to applicationmaster(ping is not considered as status > update). If your application is taking more than value configured for > "mapreduce.task.timeout", then application master consider this task as > timedout. In such case, you need to increase value for > "mapreduce.task.timeout" based on your cleanup time. > > > > > > 2. For Yarn Application, completed container's list are sent to > ApplicationMaster in heartbeat. Here you can do clean up activities for > containers. > > > > Hope this will help for you. J!! > > > > > > Thanks & Regards > > Rohith Sharma K S > > > > *From:* Krishna Kishore Bonagiri [mailto:write2kish...@gmail.com] > *Sent:* 07 April 2014 16:41 > *To:* user@hadoop.apache.org > *Subject:* Cleanup activity on YARN containers > > > > Hi, > > > > Is there any callback kind of facility, in which I can write some code > to be executed on my container at the end of my application or at the end > of that particular container execution? > > > > I want to do some cleanup activities at the end of my application, and > the clean up is not related to the localized resources that are downloaded > from HDFS. > > > > Thanks, > > Kishore > > >
