[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252876#comment-17252876 ] CruiseCheung commented on TEZ-3265: --- Agreed with [~zhangweilst], we also met this issue and it`s hard to reproduce. Added[^TEZ-3265.2.patch] to fix it, can [~sseth] to review it ? > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch, TEZ-3265.2.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17196062#comment-17196062 ] Wei Zhang commented on TEZ-3265: We are having the very much similar issue. In our case we are able to know that `perItem` is set to 2362. With maxItems = 1024 * 1024, this will lead to a primitive integer overflow of `datasize`, which equals to maxItems * perItem. With this overflowed `datasize`, `capacity < (metasize+dataSize)` will be false and the `metasize` will not be resized. > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937310#comment-15937310 ] Rajesh Balamohan commented on TEZ-3265: --- I was not able to reproduce this issue in recent releases. This can be closed for now; we can revisit this issue if it surfaces again. > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935678#comment-15935678 ] Siddharth Seth commented on TEZ-3265: - [~rajesh.balamohan] - does anything need to be done here, or can the jira be closed? > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303369#comment-15303369 ] Rajesh Balamohan commented on TEZ-3265: --- agreed. Saw this "reserved.remaining()=14680064, reserved.metasize=16777216" in one of the jobs which was weird given the checks. Increasing the mem was the easier workaround, but will check why this happened. > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3265) Add preconditions check in SortSpan when available mem is less than metasize
[ https://issues.apache.org/jira/browse/TEZ-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302550#comment-15302550 ] Siddharth Seth commented on TEZ-3265: - I'm not sure the precondition check will ever be hit - given the calculation above it. {code} if(capacity < (metasize+dataSize)) { // try to allocate less meta space, because we have sample data metasize = METASIZE*(capacity/(perItem+METASIZE)); } {code} > Add preconditions check in SortSpan when available mem is less than metasize > > > Key: TEZ-3265 > URL: https://issues.apache.org/jira/browse/TEZ-3265 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: TEZ-3265.1.patch > > > {noformat} > 2016-05-21 09:01:48,523 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 3: > reserved.remaining()=14680064, reserved.metasize=16777216 > ... > ... > ... > ... > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.position(Buffer.java:244) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.(PipelinedSorter.java:737) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:255) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:310) > at > org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283) > at > org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:198) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:542) > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:386) > ... 45 more > {noformat} > It would be good to have a Preconditions check in SortSpan instead of > throwing exception from Buffer.java. -- This message was sent by Atlassian JIRA (v6.3.4#6332)