Thanks, that was exactly what I was after!
Jakub
On 5 Nov 2014 19:41, "Sathi Chowdhury" <[email protected]> wrote:
>
> I had somewhat similar problem ,
>
> If you want to apply a memory setting to the oozie launcher job then I
> used these in the pipeline workflow xml
> <property>
> <name>oozie.launcher.mapreduce.map.memory.mb</name>
> <value>8096</value>
> </property>
> <property>
> <name>oozie.launcher.mapreduce.reduce.memory.mb</name>
> <value>8096</value>
> </property>
> <property>
> <name>oozie.launcher.mapreduce.child.java.opts</name>
> <value>-Xmx8g</value>
> </property>
>
>
>
> When I wanted apply specific memory settings to the actual java actions
> and it¹s mr run,
>
> I made my java action class take a config and keep the map reduce related
> params in it.
>
>
> hadoop jar lib/yourjavaactionclasscontainsinthis.jar
> com.x.y.JavaActionClass -libjars ${LIB_JARS} -conf yourconf.xml
> And kept required values in this config
>
> <property>
> <name>mapreduce.job.reduces</name>
> <value>49</value>
> </property>
> <property>
> <name>mapreduce.reduce.memory.mb</name>
> <value>4096</value>
> </property>
> <property>
> <name>mapreduce.reduce.java.opts</name>
> <value>-Xmx3276m</value>
> </property>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>4096</value>
> </property>
> <property>
> <name>mapreduce.map.java.opts</name>
> <value>-Xmx3276m</value>
> </property>
>
>
> (actually you can invoke this command line and see without oozie it is
> applying those settings ,then you are fine to integrate with oozie)
>
>
> A snippet of my job defn
> <main-class>com.x.y.z.parser.MyParser</main-class>
> <java-opts>-Xmx8g -Xms8g</java-opts>
> <arg>-conf</arg>
> <arg>parse-conf.xml</arg>
> <arg>-libjars</arg>
> <arg>
>
> hcatalog-core-0.12.0.2.0.11.0-1.jar,hive-metastore-0.12.0.2.0.11.0-1.jar,li
> bfb303-0.9.0.jar,hive-exec-0.12.0.2.0.11.0-1.jar,jdo-api-3.0.1.jar,libthrif
> t-0.9.0.jar,slf4j-api-1.6.6.jar
> </arg>
> <file>${workflowRoot}/lib/${JAR_FILE}#${JAR_FILE}</file>
> <file>parse-conf.xml</file>
> <capture-output/>
> </java>
>
>
> Thx and please feel free to let me know if any of it did not make sense or
> you need more info
>
>
> Then the actual map reduce job will obey these settings ,if you need any
> more
>
>
> On 11/5/14, 3:08 AM, "Jakub Stransky" <[email protected]> wrote:
>
> >Hello experienced oozie users,
> >
> >I would like to adjust hadoop settings for given coordinator/workflow job,
> >particullarily I am speaking about:
> >yarn.app.mapreduce.am.resource.mb
> >yarn.app.mapreduce.am.command-opts
> >mapreduce.map.java.opts
> >mapreduce.map.memory.mb
> >
> >Cluster defaults are to high and consume unreasonable memory amount for
> >this particular workflow. I tried to put these settings to bundle.xml for
> >a given coordinator, to coordinator
> >
>
>
