[
https://issues.apache.org/jira/browse/SPARK-60988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Zhu updated SPARK-60988:
----------------------------------
Description:
Follow-up to SPARK-58614 ([PR
#57815|https://github.com/apache/spark/pull/57815]), covering three performance
improvements:
# *Eliminate unnecessary DISTINCT processing.* Extend {{EliminateDistinct}} to
window expressions for duplicate-insensitive aggregates such as {{{}MIN{}}},
{{{}MAX{}}}, {{{}BIT_AND{}}}, {{{}BIT_OR{}}}, and {{{}COLLECT_SET{}}}. Assess
{{FIRST}} and {{LAST}} separately because of their ordering semantics.
# *Reduce full-partition position metadata.* Avoid storing
{{{}firstVisibleIndex{}}}, which is always zero for full-partition frames.
Retain {{inputIndex}} where needed to restore first-occurrence order after
sort-based deduplication.
# *Avoid redundant event sorting on the growing-frame hash path.* Currently,
every window partition creates an event sorter even when in-memory hash entries
are already in event order. This introduces substantial initialization and
allocation overhead for many small partitions. Explore direct ordered replay,
while preserving correctness under memory pressure and ensuring proper resource
cleanup. Spill-capable destructive map iteration must not be assumed to
preserve insertion order.
Add regression tests and benchmarks covering many small partitions, large
partitions, hash/sort fallback, spills, and collations. Preserve existing
aggregate semantics, memory-control guarantees, and supported window-frame
restrictions.
was:
Follow-up to #SPARK-58614 ([PR
#57815|https://github.com/apache/spark/pull/57815]), covering three performance
improvements:
# *Eliminate unnecessary DISTINCT processing.* Extend {{EliminateDistinct}} to
window expressions for duplicate-insensitive aggregates such as {{{}MIN{}}},
{{{}MAX{}}}, {{{}BIT_AND{}}}, {{{}BIT_OR{}}}, and {{{}COLLECT_SET{}}}. Assess
{{FIRST}} and {{LAST}} separately because of their ordering semantics.
# *Reduce full-partition position metadata.* Avoid storing
{{{}firstVisibleIndex{}}}, which is always zero for full-partition frames.
Retain {{inputIndex}} where needed to restore first-occurrence order after
sort-based deduplication.
# *Avoid redundant event sorting on the growing-frame hash path.* Currently,
every window partition creates an event sorter even when in-memory hash entries
are already in event order. This introduces substantial initialization and
allocation overhead for many small partitions. Explore direct ordered replay,
while preserving correctness under memory pressure and ensuring proper resource
cleanup. Spill-capable destructive map iteration must not be assumed to
preserve insertion order.
Add regression tests and benchmarks covering many small partitions, large
partitions, hash/sort fallback, spills, and collations. Preserve existing
aggregate semantics, memory-control guarantees, and supported window-frame
restrictions.
> Optimize DISTINCT window aggregate execution
> --------------------------------------------
>
> Key: SPARK-60988
> URL: https://issues.apache.org/jira/browse/SPARK-60988
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Mingliang Zhu
> Priority: Major
>
> Follow-up to SPARK-58614 ([PR
> #57815|https://github.com/apache/spark/pull/57815]), covering three
> performance improvements:
> # *Eliminate unnecessary DISTINCT processing.* Extend {{EliminateDistinct}}
> to window expressions for duplicate-insensitive aggregates such as
> {{{}MIN{}}}, {{{}MAX{}}}, {{{}BIT_AND{}}}, {{{}BIT_OR{}}}, and
> {{{}COLLECT_SET{}}}. Assess {{FIRST}} and {{LAST}} separately because of
> their ordering semantics.
> # *Reduce full-partition position metadata.* Avoid storing
> {{{}firstVisibleIndex{}}}, which is always zero for full-partition frames.
> Retain {{inputIndex}} where needed to restore first-occurrence order after
> sort-based deduplication.
> # *Avoid redundant event sorting on the growing-frame hash path.* Currently,
> every window partition creates an event sorter even when in-memory hash
> entries are already in event order. This introduces substantial
> initialization and allocation overhead for many small partitions. Explore
> direct ordered replay, while preserving correctness under memory pressure and
> ensuring proper resource cleanup. Spill-capable destructive map iteration
> must not be assumed to preserve insertion order.
> Add regression tests and benchmarks covering many small partitions, large
> partitions, hash/sort fallback, spills, and collations. Preserve existing
> aggregate semantics, memory-control guarantees, and supported window-frame
> restrictions.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
