[
https://issues.apache.org/jira/browse/TEZ-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned TEZ-1800:
-------------------------------------
Assignee: Rajesh Balamohan
> Integer overflow in ExternalSorter.getInitialMemoryRequirement()
> ----------------------------------------------------------------
>
> Key: TEZ-1800
> URL: https://issues.apache.org/jira/browse/TEZ-1800
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1800.1.patch
>
>
> When trying to implement support for tez.runtime.io.sort.mb > 2 GB in
> pipelinedsorter, I encountered integer overflow exception in ExternalSorter.
> Exception thrown is given below.
> {code}
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:158)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:97)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:192)
> at
> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:95)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:430)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:409)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Actual place where the integer overflow happens is in
> ExternalSorter.getInitialMemoryRequirement()
> {code}
> int initialMemRequestMb =
> conf.getInt(
> TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB,
> TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB_DEFAULT);
> long reqBytes = initialMemRequestMb << 20;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
