[jira] [Comment Edited] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563 ] Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:28 AM: -- [~gezapeti] * I read data from three different sources and merge them based on natural key. I have created a composite key for sorting and a natural key based on which partitioner distributes data among reducer. * My sample input data is in such a way that only one natural key is present and i have written custom partitioner to group by natural key so i should get only one input group for reducer. When I try running the MR job (*written using old API's*) as a Mapreduce action from oozie (using HUE) give me 11 input groups instead of 1. ** Reduce Input Groups 0 11 11 Reduce Input Records 0 11 11 * Running the same code using Java action from oozie (using HUE) sends only one input group to reducer ** Reduce Input Groups 0 1 1 Reduce Input Records 0 11 11 * The MR job (using old API) run using mapreduce action runs successfully and there are no errors found. The metadata has details about partitioner class but the class is never evaluated. Tried printing SysOut's from partitioner class but they don't get printed. * Attaching logs, metaData and Partitioner code for oozie MR action (using old API). Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has only one mapreduce action that reads RCFiles and creates text files. [^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml] was (Author: raghaviravi92): [~gezapeti] Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has only one mapreduce action that reads RCFiles and creates text files. [^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml] > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml > > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using >
[jira] [Comment Edited] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563 ] Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:00 AM: -- [~gezapeti] Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has only one mapreduce action that reads RCFiles and creates text files. [^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml] was (Author: raghaviravi92): [~gezapeti] Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has only one mapreduce action that reads RCFiles and creates text files. [^Logs.zip] [^PonRankPartitioner.java] > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml > > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563 ] Raghavi Ravi commented on OOZIE-3057: - [~gezapeti] Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has only one mapreduce action that reads RCFiles and creates text files. [^Logs.zip] [^PonRankPartitioner.java] > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > Attachments: Logs.zip, PonRankPartitioner.java > > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavi Ravi updated OOZIE-3057: Attachment: PonRankPartitioner.java > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > Attachments: Logs.zip, PonRankPartitioner.java > > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavi Ravi updated OOZIE-3057: Attachment: Logs.zip > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > Attachments: Logs.zip > > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168796#comment-16168796 ] Raghavi Ravi commented on OOZIE-3057: - [~rkanter] > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavi Ravi updated OOZIE-3057: Priority: Critical (was: Major) > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi >Priority: Critical > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavi Ravi updated OOZIE-3057: Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) Linux version 3.10.0-327.10.1.el7.x86_64 (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 oozie version - 4.1.0 cdh version - 5.10.1 Hue™ 3.11 - The Hadoop UI was: Red Hat Enterprise Linux Server release 7.2 (Maipo) Linux version 3.10.0-327.10.1.el7.x86_64 (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 oozie version - 4.1.0 Hue™ 3.11 - The Hadoop UI > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > cdh version - 5.10.1 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
[ https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavi Ravi updated OOZIE-3057: Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) Linux version 3.10.0-327.10.1.el7.x86_64 (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 oozie version - 4.1.0 Hue™ 3.11 - The Hadoop UI was: Red Hat Enterprise Linux Server release 7.2 (Maipo) Linux version 3.10.0-327.10.1.el7.x86_64 (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 cdh version - 5.10.1 oozie version - 4.1.0 Hue™ 3.11 - The Hadoop UI > Custom Partitioner not working in Oozie Mapreduce action > > > Key: OOZIE-3057 > URL: https://issues.apache.org/jira/browse/OOZIE-3057 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.1.0 > Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) > Linux version 3.10.0-327.10.1.el7.x86_64 > (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red > Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 > oozie version - 4.1.0 > Hue™ 3.11 - The Hadoop UI >Reporter: Raghavi Ravi > > I implemented secondary sort in mapreduce using old API > (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). > Though I have set the partitioner class in the properties, the partitioner is > not being executed. So, I'm not getting output as expected. > The same code runs fine when run using hadoop command from CLI. > And here is my workflow.xml > > > > Action failed, error > message[${wf:errorMessage(wf:lastErrorNode())}] > > > > ${jobTracker} > ${nameNode} > > > mapred.output.dir > /test_1109_3 > > > mapred.input.dir > > /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ > > > mapred.input.format.class > org.apache.hadoop.hive.ql.io.RCFileInputFormat > > > mapred.mapper.class > PonRankMapper > > > mapred.reducer.class > PonRankReducer > > > mapred.output.value.comparator.class > PonRankGroupingComparator > > > mapred.mapoutput.key.class > PonRankPair > > > mapred.mapoutput.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.output.key.class > org.apache.hadoop.io.NullWritable > > > mapred.reduce.output.value.class > org.apache.hadoop.io.Text > > > mapred.reduce.tasks > 1 > > > mapred.partitioner.class > PonRankPartitioner > > > mapred.mapper.new-api > False > > > > > > > > When running using hadoop jar command, I set the partitioner class using > JobConf.setPartitionerClass API. > Partitioner is not executed when using old API . Inspite of adding the > property. > > mapred.partitioner.class > PonRankPartitioner > > Executed the same logic using new API's (org.apache.hadoop.mapreduce) and > added mapreduce.partitioner.class property in workflow. > Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action
Raghavi Ravi created OOZIE-3057: --- Summary: Custom Partitioner not working in Oozie Mapreduce action Key: OOZIE-3057 URL: https://issues.apache.org/jira/browse/OOZIE-3057 Project: Oozie Issue Type: Bug Components: action, workflow Affects Versions: 4.1.0 Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo) Linux version 3.10.0-327.10.1.el7.x86_64 (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016 cdh version - 5.10.1 oozie version - 4.1.0 Hue™ 3.11 - The Hadoop UI Reporter: Raghavi Ravi I implemented secondary sort in mapreduce using old API (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue). Though I have set the partitioner class in the properties, the partitioner is not being executed. So, I'm not getting output as expected. The same code runs fine when run using hadoop command from CLI. And here is my workflow.xml Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}] ${jobTracker} ${nameNode} mapred.output.dir /test_1109_3 mapred.input.dir /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/ mapred.input.format.class org.apache.hadoop.hive.ql.io.RCFileInputFormat mapred.mapper.class PonRankMapper mapred.reducer.class PonRankReducer mapred.output.value.comparator.class PonRankGroupingComparator mapred.mapoutput.key.class PonRankPair mapred.mapoutput.value.class org.apache.hadoop.io.Text mapred.reduce.output.key.class org.apache.hadoop.io.NullWritable mapred.reduce.output.value.class org.apache.hadoop.io.Text mapred.reduce.tasks 1 mapred.partitioner.class PonRankPartitioner mapred.mapper.new-api False When running using hadoop jar command, I set the partitioner class using JobConf.setPartitionerClass API. Partitioner is not executed when using old API . Inspite of adding the property. mapred.partitioner.class PonRankPartitioner Executed the same logic using new API's (org.apache.hadoop.mapreduce) and added mapreduce.partitioner.class property in workflow. Partitioner was executed and desired outcome was seen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)