Hi Kishore,
Is jobs are submitted through MapReduce or Is it Yarn Application?
1. For MapReduce Framwork, framework itself provides facility to clean up
per task level.
Is there any callback kind of facility, in which I can write some
code to be executed on my container at the end of my application or at the end
of that particular container execution?
>>> You can override setup() and cleanup() for doing initialization and
>>> cleanup of your task. This facility is provided by MapReduce framework.
The call flow of task execution is
The framework first calls
setup(org.apache.hadoop.mapreduce.Mapper.Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82setup%E2%98%82org.apache.hadoop.mapreduce.Mapper.Context>,
followed by map(Object, Object,
Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82map%E2%98%82Object%E2%98%82Object%E2%98%82Context>
/ reduce(Object, Iterable,
Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BReducer.java%E2%98%83Reducer%E2%98%82%E2%98%82reduce%E2%98%82Object%E2%98%82Iterable%E2%98%82Context>
for each key/value pair. Finally
cleanup(Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82cleanup%E2%98%82Context>
is called.
Note : In clean up, do not hold container for more than
"mapreduce.task.timeout". Because, once map/reduce is completed, progress will
not be sent to applicationmaster(ping is not considered as status update). If
your application is taking more than value configured for
"mapreduce.task.timeout", then application master consider this task as
timedout. In such case, you need to increase value for "mapreduce.task.timeout"
based on your cleanup time.
2. For Yarn Application, completed container's list are sent to
ApplicationMaster in heartbeat. Here you can do clean up activities for
containers.
Hope this will help for you. :)!!
Thanks & Regards
Rohith Sharma K S
From: Krishna Kishore Bonagiri [mailto:[email protected]]
Sent: 07 April 2014 16:41
To: [email protected]
Subject: Cleanup activity on YARN containers
Hi,
Is there any callback kind of facility, in which I can write some code to be
executed on my container at the end of my application or at the end of that
particular container execution?
I want to do some cleanup activities at the end of my application, and the
clean up is not related to the localized resources that are downloaded from
HDFS.
Thanks,
Kishore
