[
https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Travis Woodruff updated TEZ-3757:
---------------------------------
Description:
This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
the {{SortSpan}} constructor:
{code}
//TODO: fix per item being passed.
span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
(1024*1024),
perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
{code}
{{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
{code}
int dataSize = maxItems * perItem;
{code}
This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
(usually?) ends up causing the capacity check to not work correctly, which
causes subsequent buffer operations to fail.
was:
This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
the {{SortSpan}} constructor:
{code}
//TODO: fix per item being passed.
span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
(1024*1024),
perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
{code}
{{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
{code}
int dataSize = maxItems * perItem;
{code}
This means that if {{perItem}} is >= 2040, {{dataSize}} overflows, which
(usually?) ends up causing the capacity check to not work correctly, which
causes subsequent buffer operations to fail.
> Integer overflow in PipelinedSorter
> -----------------------------------
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
> the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
> (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
> (usually?) ends up causing the capacity check to not work correctly, which
> causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
