[
https://issues.apache.org/jira/browse/TEZ-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-1937:
----------------------------------
Attachment: TEZ-1937.WIP.patch
Attaching WIP patch. This should be faster than the current approach. However,
If we can move EOF_MARKER out of compressed block (in IFile.close() ), then we
can just copy the compressed IFile content to another IFile.
> Reduce cost of merging ifiles in UnorderedPartitionedWriter
> -----------------------------------------------------------
>
> Key: TEZ-1937
> URL: https://issues.apache.org/jira/browse/TEZ-1937
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Attachments: TEZ-1937.WIP.patch
>
>
> Currently we iterate through all spilled files for merging. This incurs
> additional deserialization cost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
