[
https://issues.apache.org/jira/browse/TEZ-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318770#comment-17318770
]
Hadoop QA commented on TEZ-4301:
--------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} TEZ-4301 does not apply to master. Rebase required? Wrong Branch?
See https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | TEZ-4301 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13023682/TEZ-4301.2.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-TEZ-Build/106/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> TezChlid OOM cause PipelineSorter may create too many sort span
> ---------------------------------------------------------------
>
> Key: TEZ-4301
> URL: https://issues.apache.org/jira/browse/TEZ-4301
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.0, 0.9.2
> Environment:
> Reporter: CruiseCheung
> Priority: Minor
> Attachments: TEZ-4301.1.patch, TEZ-4301.2.patch
>
>
> When previous span perItem is much large than others, reserved.remaining()
> is large。This may create too many min sort span，cause now we use perItem
> length to deduce next sort span's metaSize. In production, we may have 1
> million sort span !
>
> Previous sort span:
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=965738496, reserved.metasize=16777216
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span0.length = 677, perItem =
> 1400122
> [TezChild] |impl.PipelinedSorter|: Reducer 24: done sorting span=0,
> length=677, time=0
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Heap = SpanIterator<0:676>
> (span=Span[16777216,947882594]),
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span131.length = 1, perItem =
> 442
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276247288, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span132.length = 1,
> perItem = 442, counter:533341
> [TezChild] |impl.PipelinedSorter|: Reducer 24: Span132.length = 1, perItem =
> 416
> [TezChild] |impl.PipelinedSorter|: Reducer 24:
> reserved.remaining()=276246856, reserved.metasize=16
> [TezChild] |impl.PipelinedSorter|: Reducer 24: New Span133.length = 1,
> perItem = 416, counter:533342
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
