[
https://issues.apache.org/jira/browse/PIG-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872363#action_12872363
]
Aniket Mokashi commented on PIG-282:
------------------------------------
1. It is suitable to have PARTITION BY mapreduce.Partitioner than UDF. This
will be followed by PARALLEL n.
2. Applicable to-
GROUP
COGROUP
CROSS
DISTINCT
JOIN (except 'skewed' which uses SkewedPartitioner)
3. ORDER partition by - not supported.
4. No check for validation of custom partitioners parameters
(<PigNullableWritable, Writable>).
Approach-
1. Added support for ClassType parsing and validation. Parsing for "partition
by" is added to above mentioned clauses separately.
2. Custom Partitioner is stored as a String in LO, PO and MR plan.
LogicalOperator holds the partitioner in LO plan. We add partitioner to
POGlobalRearrangement as it decides the map-reduce boundary. We read and set
the partitioner when we visit the POGlobalRearrangement.
Attaching a patch with initial changes...
> Custom Partitioner
> ------------------
>
> Key: PIG-282
> URL: https://issues.apache.org/jira/browse/PIG-282
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Reporter: Amir Youssefi
> Assignee: Aniket Mokashi
> Priority: Minor
> Fix For: 0.8.0
>
>
> By adding custom partitioner we can give control over which output partition
> a key (/value) goes to. We can add keywords to language e.g.
> PARTITION BY UDF(...)
> or a similar syntax. UDF returns a number between 0 and n-1 where n is number
> of output partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
