[
https://issues.apache.org/jira/browse/TEZ-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cyrille Chépélov updated TEZ-2256:
----------------------------------
Attachment: remove-btse-1-rfc.patch
The following patch removes BTSE and switches to a flag-based approach. It
passes the existing unit tests (which did cover at least part of the code paths)
[~hitesh], does it look like in the right direction?
Next:
# check how this ports to _master_
# check out SortedOutput as well, removing duplicate code if there's an
opportunity to do this _master_.
> Avoid use of BufferTooSmallException to signal end of buffer in
> UnorderedPartitionedKVWriter
> --------------------------------------------------------------------------------------------
>
> Key: TEZ-2256
> URL: https://issues.apache.org/jira/browse/TEZ-2256
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Cyrille Chépélov
> Assignee: Cyrille Chépélov
> Priority: Minor
> Attachments: remove-btse-1-rfc.patch
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> UnorderedPartitionedKVWriter delegates serialization to the application,
> passing it a private ByteArrayOutputStream. In case the buffer is exhausted,
> ByteArrayOutputStream signals that with a private BufferTooSmallException,
> which can be seen but not dealt with by the application. As [~cwensel]
> pointed out, when the application is in fact a complex framework, there is no
> way to distinguish this exception from a real failure, which compels logging
> the full stack even for reasonable events such as "buffer complete".
> Suggested approach: set a "complete" flag in ByteArrayOutputStream that
> disables any further output, and replace BufferTooSmallException (BTSE)
> handling by checking that flag.
> [~sseth] suggested checking out SortedOutput as well, as the mechanisms there
> should be similar.
> I'll give this a go this week.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
