[
https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17196062#comment-17196062
]
Wei Zhang commented on TEZ-3265:
--------------------------------
We are having the very much similar issue. In our case we are able to know that
`perItem` is set to 2362. With maxItems = 1024 * 1024, this will lead to a
primitive integer overflow of `datasize`, which equals to maxItems * perItem.
With this overflowed `datasize`, `capacity < (metasize+dataSize)` will be false
and the `metasize` will not be resized.
> Add preconditions check in SortSpan when available mem is less than metasize
> ----------------------------------------------------------------------------
>
> Key: TEZ-3265
> URL: https://issues.apache.org/jira/browse/TEZ-3265
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Attachments: TEZ-3265.1.patch
>
>
> {noformat}
> 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3:
> reserved.remaining()=14680064, reserved.metasize=16777216
> ...
> ...
> ...
> ...
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:244)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.<init>(PipelinedSorter.java:737)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310)
> at
> org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386)
> ... 45 more
> {noformat}
> It would be good to have a Preconditions check in SortSpan instead of
> throwing exception from Buffer.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
