Hello experienced oozie users,
I would like to adjust hadoop settings for given coordinator/workflow job,
particullarily I am speaking about:
yarn.app.mapreduce.am.resource.mb
yarn.app.mapreduce.am.command-opts
mapreduce.map.java.opts
mapreduce.map.memory.mb
Cluster defaults are to high and consume unreasonable memory amount for
this particular workflow. I tried to put these settings to bundle.xml for
a given coordinator, to coordinator
<coordinator-app name="pif-move-and-clean-final-location"
frequency="${coord:hours(1)}"
start="${start_time}" end="${end_time}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.2">
<datasets>
<include>${nameNode}/apps/dataset-definitions/datasets.xml</include>
</datasets>
<input-events>
<data-in name="categorize-output" dataset="dobCategorizeOutputRoot">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${nameNode}/apps/jobs/pif-merge-job</app-path>
<configuration>
<property>
<name>wfInputDir</name>
<value>${coord:dataIn('categorize-output')}</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx256m</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>512</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx256m</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
but so far with no luck.
The reason why I am trying to do this is that is that workflow is a simple
java action running simple operation and triggers another workflow but the
parent workflow won't finish untill trigered workflow finishes so AM and
mapper (oozie luncher for java action) blocks the memory which I would
like to minimize this way.
Hope that makes sense.
Any Idea?
