[
https://issues.apache.org/jira/browse/TEZ-2085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318948#comment-14318948
]
Hitesh Shah commented on TEZ-2085:
----------------------------------
Thanks [~rajesh.balamohan].
One more question: Does TEZ_RUNTIME_PIPELINED_SORTER_BLOCK_SIZE depend on any
environmental factors? i.e. does it need to be range bound based on available
memory or container size or capped to a certain limit? Can it exceed 2G? 2
areas I am looking to cover - if this is based on env factors, do we really
need a default value? Or the value be decided at runtime if no value is
specified? The other is whether there are range limitations and if more checks
are needed to verify this value at the start.
> PipelinedSorter should bail out (on BufferOverflowException) instead of
> retrying continuously
> ---------------------------------------------------------------------------------------------
>
> Key: TEZ-2085
> URL: https://issues.apache.org/jira/browse/TEZ-2085
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-2085.1.patch, TEZ-2085.2.patch, TEZ-2085.3.patch
>
>
> If we try to fit in a key/value pair which is great than the size that sort
> span can accommodate, PipelinedSorter would try to sort/spill indefinitely.
> This is more of a corner case. It should bail out gracefully and can throw
> back IOException instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
