[jira] [Comment Edited] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2018-02-13 Thread Raghavi Ravi (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563
 ] 

Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:28 AM:
--

[~gezapeti]
 * I read data from three different sources and merge them based on natural 
key. I have created a composite key for sorting and a natural key based on 
which partitioner distributes data among reducer.
 * My sample input data is in such a way that only one natural key is present 
and i have written custom partitioner to group by natural key so i should get 
only one input group for reducer. When I try running the MR job (*written using 
old API's*) as a Mapreduce action from oozie (using HUE) give me 11 input 
groups instead of 1.
 ** Reduce Input Groups 0 11 11
Reduce Input Records 0 11 11
 * Running the same code using Java action from oozie (using HUE) sends only 
one input group to reducer
 ** Reduce Input Groups 0 1 1
Reduce Input Records 0 11 11
 * The MR job (using old API) run using mapreduce action runs successfully and 
there are no errors found. The metadata has details about partitioner class but 
the class is never evaluated. Tried printing SysOut's from partitioner class 
but they don't get printed.
 * Attaching logs, metaData and Partitioner code for oozie MR action (using old 
API).

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]


was (Author: raghaviravi92):
[~gezapeti]

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> 

[jira] [Comment Edited] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2018-02-13 Thread Raghavi Ravi (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563
 ] 

Raghavi Ravi edited comment on OOZIE-3057 at 2/14/18 7:00 AM:
--

[~gezapeti]

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java] [^workflow.xml]


was (Author: raghaviravi92):
[~gezapeti]

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java]

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip, PonRankPartitioner.java, workflow.xml
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2018-02-13 Thread Raghavi Ravi (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363563#comment-16363563
 ] 

Raghavi Ravi commented on OOZIE-3057:
-

[~gezapeti]

Attaching oozie logs, Partitioner class and workflow.xml. The workflow.xml has 
only one mapreduce action that reads RCFiles and creates text files.

[^Logs.zip] [^PonRankPartitioner.java]

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip, PonRankPartitioner.java
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2018-02-13 Thread Raghavi Ravi (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Attachment: PonRankPartitioner.java

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip, PonRankPartitioner.java
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2018-02-13 Thread Raghavi Ravi (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Attachment: Logs.zip

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
> Attachments: Logs.zip
>
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2017-09-15 Thread Raghavi Ravi (JIRA)

[ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168796#comment-16168796
 ] 

Raghavi Ravi commented on OOZIE-3057:
-

[~rkanter] 

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2017-09-15 Thread Raghavi Ravi (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Priority: Critical  (was: Major)

> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>Priority: Critical
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2017-09-15 Thread Raghavi Ravi (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Environment: 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
cdh version - 5.10.1
Hue™ 3.11 - The Hadoop UI

  was:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI


> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> cdh version - 5.10.1
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2017-09-15 Thread Raghavi Ravi (JIRA)

 [ 
https://issues.apache.org/jira/browse/OOZIE-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghavi Ravi updated OOZIE-3057:

Environment: 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI

  was:
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
cdh version - 5.10.1
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI


> Custom Partitioner not working in Oozie Mapreduce action
> 
>
> Key: OOZIE-3057
> URL: https://issues.apache.org/jira/browse/OOZIE-3057
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 4.1.0
> Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux version 3.10.0-327.10.1.el7.x86_64 
> (mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
> Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
> oozie version - 4.1.0
> Hue™ 3.11 - The Hadoop UI
>Reporter: Raghavi Ravi
>
> I implemented secondary sort in mapreduce using old API 
> (org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).
> Though I have set the partitioner class in the properties, the partitioner is 
> not being executed. So, I'm not getting output as expected.
> The same code runs fine when run using hadoop command from CLI.
> And here is my workflow.xml
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.output.dir
> /test_1109_3
> 
> 
> mapred.input.dir
> 
> /apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/
> 
> 
> mapred.input.format.class
> org.apache.hadoop.hive.ql.io.RCFileInputFormat
> 
> 
> mapred.mapper.class
> PonRankMapper
> 
> 
> mapred.reducer.class
> PonRankReducer
> 
> 
> mapred.output.value.comparator.class
> PonRankGroupingComparator
> 
> 
> mapred.mapoutput.key.class
> PonRankPair
> 
> 
> mapred.mapoutput.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.output.key.class
> org.apache.hadoop.io.NullWritable
> 
> 
> mapred.reduce.output.value.class
> org.apache.hadoop.io.Text
> 
> 
> mapred.reduce.tasks
> 1
> 
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> 
> mapred.mapper.new-api
> False
> 
> 
> 
> 
> 
> 
> 
> When running using hadoop jar command, I set the partitioner class using 
> JobConf.setPartitionerClass API.
>  Partitioner is not executed when using old API . Inspite of adding the 
> property.
> 
> mapred.partitioner.class
> PonRankPartitioner
> 
> Executed the same logic using new API's (org.apache.hadoop.mapreduce) and 
> added mapreduce.partitioner.class property in workflow.
> Partitioner was executed and desired outcome was seen.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (OOZIE-3057) Custom Partitioner not working in Oozie Mapreduce action

2017-09-15 Thread Raghavi Ravi (JIRA)
Raghavi Ravi created OOZIE-3057:
---

 Summary: Custom Partitioner not working in Oozie Mapreduce action
 Key: OOZIE-3057
 URL: https://issues.apache.org/jira/browse/OOZIE-3057
 Project: Oozie
  Issue Type: Bug
  Components: action, workflow
Affects Versions: 4.1.0
 Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Linux version 3.10.0-327.10.1.el7.x86_64 
(mockbu...@x86-021.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red 
Hat 4.8.5-4) (GCC) ) #1 SMP Sat Jan 23 04:54:55 EST 2016
cdh version - 5.10.1
oozie version - 4.1.0
Hue™ 3.11 - The Hadoop UI
Reporter: Raghavi Ravi


I implemented secondary sort in mapreduce using old API 
(org.apache.hadoop.mapred.*) and trying to execute it using Oozie (From Hue).

Though I have set the partitioner class in the properties, the partitioner is 
not being executed. So, I'm not getting output as expected.

The same code runs fine when run using hadoop command from CLI.

And here is my workflow.xml




Action failed, error 
message[${wf:errorMessage(wf:lastErrorNode())}]



${jobTracker}
${nameNode}


mapred.output.dir
/test_1109_3


mapred.input.dir

/apps/hive/warehouse/7360_0609_rx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0609_tx/day=06-09-2017/hour=13/quarter=2/,/apps/hive/warehouse/7360_0509_util/day=05-09-2017/hour=16/quarter=1/


mapred.input.format.class
org.apache.hadoop.hive.ql.io.RCFileInputFormat


mapred.mapper.class
PonRankMapper


mapred.reducer.class
PonRankReducer


mapred.output.value.comparator.class
PonRankGroupingComparator


mapred.mapoutput.key.class
PonRankPair


mapred.mapoutput.value.class
org.apache.hadoop.io.Text


mapred.reduce.output.key.class
org.apache.hadoop.io.NullWritable


mapred.reduce.output.value.class
org.apache.hadoop.io.Text


mapred.reduce.tasks
1


mapred.partitioner.class
PonRankPartitioner


mapred.mapper.new-api
False








When running using hadoop jar command, I set the partitioner class using 
JobConf.setPartitionerClass API.

 Partitioner is not executed when using old API . Inspite of adding the 
property.


mapred.partitioner.class
PonRankPartitioner


Executed the same logic using new API's (org.apache.hadoop.mapreduce) and added 
mapreduce.partitioner.class property in workflow.

Partitioner was executed and desired outcome was seen.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)