CruiseCheung created TEZ-4301:
---------------------------------
Summary: TezChlid OOM cause PipelineSorter may create too many
sort span
Key: TEZ-4301
URL: https://issues.apache.org/jira/browse/TEZ-4301
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.2, 0.9.0
Environment: Previous sort span:
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=965738496,
reserved.metasize=16777216
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
1400122
[TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0, length=677,
time=0
[TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
(span=Span[16777216,947882594]),
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem = 442
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276247288,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1, perItem
= 442, counter:533341
[TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
416
[TezChild] |impl.PipelinedSorter|: Reducer 24: reserved.remaining()=276246856,
reserved.metasize=16
[TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1, perItem
= 416, counter:533342
Reporter: CruiseCheung
When previous span perItem is much large than others, reserved.remaining() is
large。This may create too many min sort span，cause now we use perItem length
to deduce next sort span's metaSize.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
