[
https://issues.apache.org/jira/browse/PIG-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-3435:
------------------------------
Attachment: pig-3435-v02_skipcustompatitioner_for_merge.patch
While looking at the testcase, found PIG-2627 where it fixed one of the issues
with custom-partitioner and multiquery optimization (but not all).
Specific case mentioned on that ticket is handled on that jira and it works,
but my patch here simply skips multiquery optimization for ALL custom
partitioner jobs.
Since it's sort of a correctness issue, I want this fix to be back-ported to
0.11. And for that, I kept the change to be simple.
Can we create a separate jira for reviving custom-partitioner + multiquery
optimization for later releases?
> Custom Partitioner not working with MultiQueryOptimizer
> -------------------------------------------------------
>
> Key: PIG-3435
> URL: https://issues.apache.org/jira/browse/PIG-3435
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Attachments: pig-3435-v01.patch,
> pig-3435-v02_skipcustompatitioner_for_merge.patch
>
>
> When looking at PIG-3385, noticed some issues in handling of custom
> partitioner with multi-query optimization.
> {noformat}
> C1 = group B1 by col1 PARTITION BY
> org.apache.pig.test.utils.SimpleCustomPartitioner parallel 2;
> C2 = group B2 by col1 PARTITION BY
> org.apache.pig.test.utils.SimpleCustomPartitioner parallel 2;
> {noformat}
> This seems to be merged to one mapreduce job correctly but custom partitioner
> information was lost.
> {noformat}
> C1 = group B1 by col1 PARTITION BY
> org.apache.pig.test.utils.SimpleCustomPartitioner parallel 2;
> C2 = group B2 by col1 parallel 2;
> {noformat}
> This seems to be merged even though they should run on two different
> partitioner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
