[
https://issues.apache.org/jira/browse/SPARK-59941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Tenedorio resolved SPARK-59941.
--------------------------------------
Fix Version/s: 4.3.1
4.4.0
Resolution: Fixed
Issue resolved by pull request 59248
[https://github.com/apache/spark/pull/59248]
> Widen UnsafeExternalSorter row accounting to Long for window partitions
> larger than Int.MaxValue rows
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-59941
> URL: https://issues.apache.org/jira/browse/SPARK-59941
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Mark Jarvin
> Assignee: Mark Jarvin
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.1, 4.4.0
>
>
> h2. Summary
> {{org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter}}
> accounts for buffered rows
> in 32-bit {{int}}s on the read path:
> * {{UnsafeSorterSpillWriter.recordsSpilled()}} returns {{int}}.
> * {{UnsafeSorterIterator.getNumRecords()}} returns {{int}}.
> * {{UnsafeInMemorySorter.numRecords()}} returns {{int}}.
> * {{getIterator(int startIndex)}} accumulates spilled counts into an {{int
> i}} and skips a spill
> file unless {{i + spillWriter.recordsSpilled() > startIndex}};
> {{ChainedIterator.numRecords}} is
> also {{int}}.
> When a single buffered partition exceeds {{Int.MaxValue}} rows and has
> spilled, the running {{int}}
> total overflows negative. Later spill files are then skipped (rows silently
> dropped), or
> {{moveOver}} throws a raw {{ArrayIndexOutOfBoundsException}}. This affects
> window execution even on
> the {{getIterator(0)}} path used by LEAD/LAG and unbounded-offset
> {{nth_value}}.
> h2. Context
> Follow-up to SPARK-59929, which widened the window row counter
> ({{ExternalAppendOnlyUnsafeRowArray.numRows}}/{{length}}) to {{Long}} and
> added a fail-fast guard
> ({{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}}) that currently rejects
> *every* window frame over
> a partition larger than {{Int.MaxValue}} rows. Those frames cannot be made
> correct over such
> partitions until this sorter accounting is widened.
> h2. Proposed work
> Widen the sorter's row accounting to {{long}} (the {{getIterator}}
> accumulator and skip condition,
> {{ChainedIterator.numRecords}}, and the {{recordsSpilled()}} /
> {{getNumRecords()}} /
> {{numRecords()}} surface and their consumers), then relax the window guard so
> the offset frames
> (LEAD/LAG, unbounded {{nth_value}}) produce correct results for partitions
> larger than
> {{Int.MaxValue}} rows instead of failing.
> Related: SPARK-59929, SPARK-51757.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
