[
https://issues.apache.org/jira/browse/TEZ-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated TEZ-3605:
-----------------------------
Description:
Analogous to the Unordered case we should not have empty partition
entries/segments in the Ordered/DefaultSorter case. This will save writing
unnecessary data.
Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
job, this change would allow not fetching empty partitions and then throwing
them away.
was:Analogous to the Unordered case we should not have empty partition
entries/segments in the Ordered/DefaultSorter case. This will save writing
unnecessary data.
> Detect and prune empty partitions for the Ordered case
> ------------------------------------------------------
>
> Key: TEZ-3605
> URL: https://issues.apache.org/jira/browse/TEZ-3605
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: TEZ-3605.001.patch, TEZ-3605.002.patch,
> TEZ-3605.003.patch, TEZ-3605.004.patch, TEZ-3605.005.patch, TEZ-3605.006.patch
>
>
> Analogous to the Unordered case we should not have empty partition
> entries/segments in the Ordered/DefaultSorter case. This will save writing
> unnecessary data.
> Additionally, with tez_shuffle feature (TEZ-3334), in a heavily auto reduced
> job, this change would allow not fetching empty partitions and then throwing
> them away.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
