[
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563
]
Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:28 AM:
--
[~gezapeti]
* I read data from three different sources and merge them based on natural
key. I have created a composite key for sorting and a natural key based on
which partitioner distributes data among reducer.
* My sample input data is in such a way that only one natural key is present
and i have written custom partitioner to group by natural key so i should get
only one input group for reducer. When I try running the MR job (*written using
old API's*) as a Mapreduce action from oozie (using HUE) give me 11 input
groups instead of 1.
** Reduce Input Groups 0 11 11
Reduce Input Records 0 11 11
* Running the same code using Java action from oozie (using HUE) sends only
one input group to reducer
** Reduce Input Groups 0 1 1
Reduce Input Records 0 11 11
* The MR job (using old API) run using mapreduce action runs successfully and
there are no errors found. The metadata has details about partitioner class but
the class is never evaluated. Tried printing SysOut's from partitioner class
but they don't get printed.
* Attaching logs, metaData and Partitioner code for oozie MR action (using old
API).
Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has
only one mapreduce action that reads RCFiles and creates text files.
[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]
was (Author: raghaviravi92):
[~gezapeti]
Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has
only one mapreduce action that reads RCFiles and creates text files.
[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]
> Custom Partitioner not working in Oozie Mapreduce action
>
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml
>
>
> I implemented secondary sort in mapreduce using old API
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
>
>
>
> Action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]
>
>
>
> ${jobTracker}
> ${nameNode}
>
>
> mapred.output.dir
> /test_1109_3
>
>
> mapred.input.dir
>
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
>
>
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
>
>
> mapred.mapper.class
> PonRankMapper
>
>
> mapred.reducer.class
> PonRankReducer
>
>
> mapred.output.value.comparator.class
> PonRankGroupingComparator
>
>
> mapred.mapoutput.key.class
> PonRankPair
>
>
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
>
>
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
>
>
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
>
>
> mapred.reduce.tasks
> 1
>
>
> mapred.partitioner.class
> PonRankPartitioner
>
>
> mapred.mapper.new-api
> False
>
>
>
>
>
>
>
> When running using hadoop jar command, I set the partitioner class using
>
