[
https://issues.apache.org/jira/browse/TEZ-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350196#comment-14350196
]
shanaka commented on TEZ-2105:
------------------------------
Thanks Jeff for pointing me a place to start, It's really helpful.
> Totally Sorted Edge with auto-parallelism
> -----------------------------------------
>
> Key: TEZ-2105
> URL: https://issues.apache.org/jira/browse/TEZ-2105
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Gopal V
> Labels: gsoc, gsoc2015, hadoop, java, pig, tez
>
> Pig-on-Tez supports an edge configuration using a sampled Output along with a
> vertex manager for automatic parallelism estimation.
> This is referred to in the Pig-on-Tez Hadoop Summit presentation.
> http://www.slideshare.net/Hadoop_Summit/pig-on-tez-low-latency-etl-with-big-data/19
> Migrating that plan-model into Tez as a native edge type would allow for much
> more efficient scheduling of the downstream edges and effectively turn the
> auto-parallelism implementation into a runtime skew-correcting mechanism
> within this edge.
> The Tez Edge has enough information to sample, determine partitioning order
> and correct parallelism.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
