[
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168796#comment-16168796
]
Raghavi Ravi commented on OOZIE-3057:
-------------------------------------
[~rkanter]
> Custom Partitioner not working in Oozie Mapreduce action
> --------------------------------------------------------
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
> Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
> Reporter: Raghavi Ravi
> Priority: Critical
>
> I implemented secondary sort in mapreduce using old API
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> <workflow-app name="MyTriplets" xmlns="uri:oozie:workflow:0.5">
> <start to="mapreduce-598d"/>
> <kill name="Kill">
> <message>Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <action name="mapreduce-598d">
> <map-reduce>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.output.dir</name>
> <value>/test_1109_3</value>
> </property>
> <property>
> <name>mapred.input.dir</name>
>
> <value>/apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/</value>
> </property>
> <property>
> <name>mapred.input.format.class</name>
> <value>org.apache.hadoop.hive.ql.io.RCFileInputFormat</value>
> </property>
> <property>
> <name>mapred.mapper.class</name>
> <value>PonRankMapper</value>
> </property>
> <property>
> <name>mapred.reducer.class</name>
> <value>PonRankReducer</value>
> </property>
> <property>
> <name>mapred.output.value.comparator.class</name>
> <value>PonRankGroupingComparator</value>
> </property>
> <property>
> <name>mapred.mapoutput.key.class</name>
> <value>PonRankPair</value>
> </property>
> <property>
> <name>mapred.mapoutput.value.class</name>
> <value>org.apache.hadoop.io.Text</value>
> </property>
> <property>
> <name>mapred.reduce.output.key.class</name>
> <value>org.apache.hadoop.io.NullWritable</value>
> </property>
> <property>
> <name>mapred.reduce.output.value.class</name>
> <value>org.apache.hadoop.io.Text</value>
> </property>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> </property>
> <property>
> <name>mapred.partitioner.class</name>
> <value>PonRankPartitioner</value>
> </property>
> <property>
> <name>mapred.mapper.new-api</name>
> <value>False</value>
> </property>
> </configuration>
> </map-reduce>
> <ok to="End"/>
> <error to="Kill"/>
> </action>
> <end name="End"/>
> When running using hadoop jar command, I set the partitioner class using
> JobConf.setPartitionerClass API.
> Partitioner is not executed when using old API . Inspite of adding the
> property.
> <property>
> <name>mapred.partitioner.class</name>
> <value>PonRankPartitioner</value>
> </property>
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
