[
https://issues.apache.org/jira/browse/TEZ-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074630#comment-14074630
]
Rohini Palaniswamy edited comment on TEZ-1107 at 7/25/14 5:30 PM:
------------------------------------------------------------------
[~daijy]
I suppose this does not require any API changes.
was (Author: rohini):
@daijy,
I suppose this does not require any API changes.
> Support increase of parallelism of vertex in case of custom partitioner
> -----------------------------------------------------------------------
>
> Key: TEZ-1107
> URL: https://issues.apache.org/jira/browse/TEZ-1107
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Daniel Dai
> Assignee: Bikas Saha
>
> Current VertexManagerPlugin/EdgeManager mechanism support decrease of
> parallelism of a vertex, but increase parallelism is not supported. In
> general, we need to do repartition to increase the parallelism. However, in
> my simplified case, the proceeding vertex is using a custom partitioner which
> is able to partition to the final parallelism, repartitioning is not needed.
> However, I hit an exception from sorter:
> : Caused by: java.io.IOException: Illegal partition for
> Null: false index: 0 53.8 (2), TotalPartitions: 2
> : at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:208)
> : at
> org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
>
> : at
> org.apache.tez.runtime.library.output.OnFileSortedOutput$1.write(OnFileSortedOutput.java:111)
>
> : at
> org.apache.pig.backend.hadoop.executionengine.tez.POIdentityInOutTez.getNextTuple(POIdentityInOutTez.java:148)
> : ... 8 more
> While increase parallelism in general is harder, increase parallelism with a
> custom partitioner might be easier to fix.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
