[
https://issues.apache.org/jira/browse/PIG-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-4079:
-------------------------------
Description:
I want to debug my custom partitioner in local mode. But it's not possible with
Pig because:
# MR does not invoke partitioner if # of reducers is equal to 1.
https://issues.apache.org/jira/browse/MAPREDUCE-1287
# Pig always forces a single reducers in local mode.
{code:title=LogicalPlanBuilder.java}
void setParallel(LogicalRelationalOperator op, Integer parallel) {
if( parallel != null ) {
op.setRequestedParallelism( pigContext.getExecType() ==
ExecType.LOCAL ? 1 : parallel );
}
}
{code}
After I change Pig to honor the parallel clause, my custom partitioner gets
invoked in local mode.
was:
I want to debug my custom partitioner in local mode. But it's not possible with
Pig because:
# MR does not invoke partitioner if # of reducers is equal to 1.
# Pig always forces a single reducers in local mode.
{code:title=LogicalPlanBuilder.java}
void setParallel(LogicalRelationalOperator op, Integer parallel) {
if( parallel != null ) {
op.setRequestedParallelism( pigContext.getExecType() ==
ExecType.LOCAL ? 1 : parallel );
}
}
{code}
After I change Pig to honor the parallel clause, my custom partitioner gets
invoked in local mode.
> Parallel clause is not honored in local mode
> --------------------------------------------
>
> Key: PIG-4079
> URL: https://issues.apache.org/jira/browse/PIG-4079
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.14.0
>
>
> I want to debug my custom partitioner in local mode. But it's not possible
> with Pig because:
> # MR does not invoke partitioner if # of reducers is equal to 1.
> https://issues.apache.org/jira/browse/MAPREDUCE-1287
> # Pig always forces a single reducers in local mode.
> {code:title=LogicalPlanBuilder.java}
> void setParallel(LogicalRelationalOperator op, Integer parallel) {
> if( parallel != null ) {
> op.setRequestedParallelism( pigContext.getExecType() ==
> ExecType.LOCAL ? 1 : parallel );
> }
> }
> {code}
> After I change Pig to honor the parallel clause, my custom partitioner gets
> invoked in local mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
