For every action which launches on cluster will have a launcher job. The actions like email,fs etc runs on oozie server only. Capturing output will not cost memory issue. I am not seeing that you are doing any memory intensive work on shell action, Your launcher job must be using not more than 250 mb memory. How are you verifying it? You can increase the memory of launcher job, no issue with that.
Regards, Jaydeep On Fri, Mar 18, 2016 at 12:05 PM, Nitin Kumar <[email protected]> wrote: > Hi! > > I needed some clarifications regarding the oozie launcher job. > > 1) Is the launcher job launched per workflow application (with several > actions) or per action within a workflow application? > > 2) Use Case: I have workflows that contain multiple shell actions (which > internally execute spark, hive, pig actions etc.). The reason for using > shell is because additional parameters like partition date can be computed > using custom logic and passed to hive using .q files > > Example: > > Shell file > hive -hiveconf DATABASE_NAME=$1 -hiveconf MASTER_TABLE_NAME=$2 -hiveconf > SOURCE_TABLE_NAME=$3 -hiveconf -f $4 > > q file > use ${hiveconf:DATABASE_NAME}; > insert overwrite into table ${hiveconf:MASTER_TABLE_NAME} select * from > ${hiveconf:SOURCE_TABLE_NAME}; > > I set the oozie.launcher.mapreduce.job.queuename and > mapreduce.job.queuename to different queues to avoid starvation of task > slots in a single queue. I also omitted the > <capture-output></capture-output> in the corresponding shell action. > However, I still see the launcher job occupying a lot of memory from the > launcher queue. > > - Is this because the launcher job caches the log ouput that comes from > hive? > - Is it necessary to give the launcher job enough memory when executing a > shell action the way I am? > - What would happen if I explicitly limited the launcher job memory? > > I would highly appreciate it if someone could outline the responsibilities > of the oozie launcher job. > > Thanks and Regards, > Niti > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
