[
https://issues.apache.org/jira/browse/TEZ-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315552#comment-17315552
]
CruiseCheung commented on TEZ-4301:
-----------------------------------
Added a patch for this issue. Anybody help to review the code?
> TezChlid OOM cause PipelineSorter may create too many sort span
> ---------------------------------------------------------------
>
> Key: TEZ-4301
> URL: https://issues.apache.org/jira/browse/TEZ-4301
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Environment:
> Reporter: CruiseCheung
> Priority: Minor
> Attachments: TEZ-4301.1.patch
>
>
> When previous span perItem is much large than others, reserved.remaining()
> is large。This may create too many min sort span，cause now we use perItem
> length to deduce next sort span's metaSize. In production, we may have 1
> million sort span !
>
> Previous sort span:
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=965738496, reserved.metasize=16777216
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
> 1400122
> [TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0,
> length=677, time=0
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
> (span=Span[16777216,947882594]),
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem =
> 442
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276247288, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1,
> perItem = 442, counter:533341
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
> 416
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276246856, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1,
> perItem = 416, counter:533342
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
