[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363563#comment-16363563
 ] 

Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:28 AM:
--------------------------------------------------------------

[~gezapeti]
 * I read data from three different sources and merge them based on natural 
key. I have created a composite key for sorting and a natural key based on 
which partitioner distributes data among reducer.
 * My sample input data is in such a way that only one natural key is present 
and i have written custom partitioner to group by natural key so i should get 
only one input group for reducer. When I try running the MR job (*written using 
old API's*) as a Mapreduce action from oozie (using HUE) give me 11 input 
groups instead of 1.
 ** Reduce Input Groups 0 11 11
Reduce Input Records 0 11 11
 * Running the same code using Java action from oozie (using HUE) sends only 
one input group to reducer
 ** Reduce Input Groups 0 1 1
Reduce Input Records 0 11 11
 * The MR job (using old API) run using mapreduce action runs successfully and 
there are no errors found. The metadata has details about partitioner class but 
the class is never evaluated. Tried printing SysOut's from partitioner class 
but they don't get printed.
 * Attaching logs, metaData and Partitioner code for oozie MR action (using old 
API).

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]


was (Author: raghaviravi92):
[~gezapeti]

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]

> Custom Partitioner not working in Oozie Mapreduce action
> --------------------------------------------------------
>
>                 Key: OOZIE-3057
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3057
>             Project: Oozie
>          Issue Type: Bug
>          Components: action, workflow
>    Affects Versions: 4.1.0
>         Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>            Reporter: Raghavi Ravi
>            Priority: Critical
>         Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> <workflow-app name="MyTriplets" xmlns="uri:oozie:workflow:0.5">
> <start to="mapreduce-598d"/>
> <kill name="Kill">
>     <message>Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <action name="mapreduce-598d">
>     <map-reduce>
>         <job-tracker>${jobTracker}</job-tracker>
>         <name-node>${nameNode}</name-node>
>         <configuration>
>             <property>
>                 <name>mapred.output.dir</name>
>                 <value>/test_1109_3</value>
>             </property>
>             <property>
>                 <name>mapred.input.dir</name>
>                 
> <value>/apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/</value>
>             </property>
>             <property>
>                 <name>mapred.input.format.class</name>
>                 <value>org.apache.hadoop.hive.ql.io.RCFileInputFormat</value>
>             </property>
>             <property>
>                 <name>mapred.mapper.class</name>
>                 <value>PonRankMapper</value>
>             </property>
>             <property>
>                 <name>mapred.reducer.class</name>
>                 <value>PonRankReducer</value>
>             </property>
>             <property>
>                 <name>mapred.output.value.comparator.class</name>
>                 <value>PonRankGroupingComparator</value>
>             </property>
>             <property>
>                 <name>mapred.mapoutput.key.class</name>
>                 <value>PonRankPair</value>
>             </property>
>             <property>
>                 <name>mapred.mapoutput.value.class</name>
>                 <value>org.apache.hadoop.io.Text</value>
>             </property>
>             <property>
>                 <name>mapred.reduce.output.key.class</name>
>                 <value>org.apache.hadoop.io.NullWritable</value>
>             </property>
>             <property>
>                 <name>mapred.reduce.output.value.class</name>
>                 <value>org.apache.hadoop.io.Text</value>
>             </property>
>             <property>
>                 <name>mapred.reduce.tasks</name>
>                 <value>1</value>
>             </property>
>             <property>
>                 <name>mapred.partitioner.class</name>
>                 <value>PonRankPartitioner</value>
>             </property>
>             <property>
>                 <name>mapred.mapper.new-api</name>
>                 <value>False</value>
>             </property>
>         </configuration>
>     </map-reduce>
>     <ok to="End"/>
>     <error to="Kill"/>
> </action>
> <end name="End"/>
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
>             <property>
>                 <name>mapred.partitioner.class</name>
>                 <value>PonRankPartitioner</value>
>             </property>
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to