[
https://issues.apache.org/jira/browse/PIG-539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649436#action_12649436
]
Olga Natkovich commented on PIG-539:
------------------------------------
The current semantics of PARALLEL is that it controls the number of reducers
and not the number of maps. The number of maps is controlled by the slicer. You
might need a custom slicer to make sure that all your data is flowing through a
single map.
To change the semantics, we need to have a wide discussion on the user list and
get community support for this.
> unable to control parallelism of Map tasks
> ------------------------------------------
>
> Key: PIG-539
> URL: https://issues.apache.org/jira/browse/PIG-539
> Project: Pig
> Issue Type: Bug
> Components: impl
> Environment: local execution + hadoop execution
> Reporter: Christopher Olston
>
> I put "PARALLEL 1" following *every* statement in my pig script, and it still
> executes maps with more than 1 parallel task. This is a major problem because
> for one of my operations I need to have a serialized (non-parallel) map.
> Probably the semantics of parallelism should be as follows:
> 1. group pig operators into map/reduce stages
> 2. for each stage, take the minimum of the "Parallel" directives given by
> the user for statements executed as part of that stage
> (We'll have to decide on a rule for statements that use the combiner, which
> execute partially on the map side and partially on the reduce side ...)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
