[
https://issues.apache.org/jira/browse/SPARK-56896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56896:
---------------------------------
Description:
Replaces per-element {{readValue}} loops with two-pass bulk read + in-place
conversion for timestamp and date Parquet vector updaters.
h3. Changes
* *Bulk read + convert*: For each updater, first read all raw long/int values
in a single batch call, then convert in-place (e.g. rebase,
microseconds-to-nanoseconds).
* *Updaters optimized*: {{LongAsMicrosUpdater}}, {{LongAsNanosUpdater}},
{{LongAsMicrosRebaseUpdater}}, {{DateToTimestampNTZWithRebaseUpdater}}.
* Avoids per-element virtual dispatch through {{VectorizedValuesReader}}.
Note: {{DateToTimestampNTZUpdater}} (CORRECTED mode) was already optimized via
SPARK-56804.
h3. Benchmark Results (GHA, AMD EPYC 7763)
||Updater||Speedup (JDK 17 / JDK 21 / JDK 25)||
|LongAsMicrosUpdater|2.1x / 2.9x / 3.3x|
|LongAsNanosUpdater|~2.6x (local runs)|
|LongAsMicrosRebaseUpdater|~2.1x (local runs)|
|DateToTimestampNTZWithRebaseUpdater|~2.0x (local runs)|
PR: https://github.com/apache/spark/pull/55923
Parent issue: https://github.com/apache/spark/issues/56011
> Add bulk read paths for timestamp/date Parquet vector updaters
> --------------------------------------------------------------
>
> Key: SPARK-56896
> URL: https://issues.apache.org/jira/browse/SPARK-56896
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Replaces per-element {{readValue}} loops with two-pass bulk read + in-place
> conversion for timestamp and date Parquet vector updaters.
> h3. Changes
> * *Bulk read + convert*: For each updater, first read all raw long/int values
> in a single batch call, then convert in-place (e.g. rebase,
> microseconds-to-nanoseconds).
> * *Updaters optimized*: {{LongAsMicrosUpdater}}, {{LongAsNanosUpdater}},
> {{LongAsMicrosRebaseUpdater}}, {{DateToTimestampNTZWithRebaseUpdater}}.
> * Avoids per-element virtual dispatch through {{VectorizedValuesReader}}.
> Note: {{DateToTimestampNTZUpdater}} (CORRECTED mode) was already optimized
> via SPARK-56804.
> h3. Benchmark Results (GHA, AMD EPYC 7763)
> ||Updater||Speedup (JDK 17 / JDK 21 / JDK 25)||
> |LongAsMicrosUpdater|2.1x / 2.9x / 3.3x|
> |LongAsNanosUpdater|~2.6x (local runs)|
> |LongAsMicrosRebaseUpdater|~2.1x (local runs)|
> |DateToTimestampNTZWithRebaseUpdater|~2.0x (local runs)|
> PR: https://github.com/apache/spark/pull/55923
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
