[
https://issues.apache.org/jira/browse/SPARK-55318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhidong Qu updated SPARK-55318:
-------------------------------
Description:
Implement a set of performance optimizations for the vector aggregation
functions recently added.
* Reuse binary buffer in-place: instead of allocating a new ByteBuffer for
each update/merge call
* Hoist division out of loop: compute invCount = 1.0f / newCount once before
the loop instead of dividing per element
* Hoist weight calculations out of loop: compute leftWeight and rightWeight
once before the loop instead of 2 divisions per element
* Skip null checks when unnecessary: Check ArrayType.containsNull at
initialization and skip the per-element null check entirely when the array type
cannot contain nulls
> Performance Optimizations for vector_sum, vector_avg
> ----------------------------------------------------
>
> Key: SPARK-55318
> URL: https://issues.apache.org/jira/browse/SPARK-55318
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Zhidong Qu
> Assignee: Zhidong Qu
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> Implement a set of performance optimizations for the vector aggregation
> functions recently added.
> * Reuse binary buffer in-place: instead of allocating a new ByteBuffer for
> each update/merge call
> * Hoist division out of loop: compute invCount = 1.0f / newCount once before
> the loop instead of dividing per element
> * Hoist weight calculations out of loop: compute leftWeight and rightWeight
> once before the loop instead of 2 divisions per element
> * Skip null checks when unnecessary: Check ArrayType.containsNull at
> initialization and skip the per-element null check entirely when the array
> type cannot contain nulls
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
