Mark Jarvin created SPARK-60269:
-----------------------------------
Summary: Support or document window partitions larger than
Int.MaxValue rows for the remaining window frames
Key: SPARK-60269
URL: https://issues.apache.org/jira/browse/SPARK-60269
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Mark Jarvin
h2. Summary
After SPARK-59929 and SPARK-59941, LEAD/LAG and NTH_VALUE over the whole
partition return correct results for partitions with more than {{Int.MaxValue}}
rows. Every other window frame still fails fast with
{{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}} because it tracks the row
index or window bounds in 32-bit {{Int}}s. This ticket tracks what it would
take to support them, and which ones should stay rejected. It is low priority:
it only affects workloads with a single partition over 2^31 rows, and the
current behaviour is a clear error, not wrong results.
h2. Sub-items
*1. Plain aggregate frames (most useful).* Whole-partition aggregates (for
example {{sum(x) OVER (PARTITION BY k)}}) and running aggregates (UNBOUNDED
PRECEDING to CURRENT ROW). Needs:
* {{WindowFunctionFrame.write(index: Int, ...)}} and the drivers' {{rowIndex}}
widened to {{Long}} ({{WindowEvaluatorFactory}}).
* {{BoundOrdering.compare}} taking {{Long}} indexes.
* {{AggregateProcessor.initialize(size: Int)}} taking the partition size as
{{Long}}.
Sliding frames additionally need the frame bounds widened, and can follow.
*2. Size-based functions: NTILE, CUME_DIST, PERCENT_RANK.* They keep the
partition size in an {{INT}} aggregate buffer column
({{SizeBasedWindowFunction.n}}), so this needs a buffer schema change. Separate
from item 1.
*3. Frames and functions that should stay rejected (document, improve the
message).*
* {{ROW_NUMBER}}, {{RANK}}, {{DENSE_RANK}} return {{INT}}, so a result above
{{Int.MaxValue}} cannot be represented.
* UNBOUNDED FOLLOWING frames recompute the aggregate for every row (quadratic)
and use {{getIterator(int startIndex)}}; not usable at this size.
* The Python (Arrow) window path sends the window bounds to Python as 32-bit
ints.
* {{WindowSegmentTree}} indexes rows with {{Int}}s.
The deliverable here is a clearer error message and a documentation note, not
code.
Related: SPARK-59929, SPARK-59941.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
