[
https://issues.apache.org/jira/browse/SPARK-56892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56892:
---------------------------------
Description:
Replaces per-element lambda dispatch in {{readIntegers}}/{{readLongs}} with
bulk paths that compute prefix sums in-place and write via
{{putInts}}/{{putLongs}} in the DELTA_BINARY_PACKED vectorized Parquet decoder
({{VectorizedDeltaBinaryPackedReader}}).
h3. Changes
* *Bulk INT32/INT64 reads*: Replace per-value lambda callbacks with in-place
prefix-sum computation followed by a single {{putInts}}/{{putLongs}} call.
* *Bulk INT32/INT64 skip*: Replace per-value skip with batch prefix-sum
computation (values discarded).
* *readUnsignedLongs*: Eliminate 3 allocations per value by replacing
{{BigInteger(Long.toUnsignedString(v))}} with a zero-allocation {{byte[]}} loop
encoder.
* *Widening overrides*: Add {{readIntegersAsLongs}} and
{{readIntegersAsDoubles}} that skip the int narrowing step entirely.
h3. Benchmark Results (GHA, AMD EPYC 7763, JDK 17/21/25)
||Type||Speedup||
|INT32 reads|1.1-1.6x|
|INT32 skip|1.3-1.8x|
|INT64 reads|1.8-3.7x|
|INT64 skip|2.3-4.0x|
|readIntegersAsLongs (INT32 -> Long)|2.4-2.7x|
|readIntegersAsDoubles (INT32 -> Double)|2.1-2.4x|
|readUnsignedLongs|7.3-8.2x|
PR: https://github.com/apache/spark/pull/55919
Parent issue: https://github.com/apache/spark/issues/56011
> Bulk read optimization for Parquet DELTA_BINARY_PACKED decoding
> ---------------------------------------------------------------
>
> Key: SPARK-56892
> URL: https://issues.apache.org/jira/browse/SPARK-56892
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Replaces per-element lambda dispatch in {{readIntegers}}/{{readLongs}} with
> bulk paths that compute prefix sums in-place and write via
> {{putInts}}/{{putLongs}} in the DELTA_BINARY_PACKED vectorized Parquet
> decoder ({{VectorizedDeltaBinaryPackedReader}}).
> h3. Changes
> * *Bulk INT32/INT64 reads*: Replace per-value lambda callbacks with in-place
> prefix-sum computation followed by a single {{putInts}}/{{putLongs}} call.
> * *Bulk INT32/INT64 skip*: Replace per-value skip with batch prefix-sum
> computation (values discarded).
> * *readUnsignedLongs*: Eliminate 3 allocations per value by replacing
> {{BigInteger(Long.toUnsignedString(v))}} with a zero-allocation {{byte[]}}
> loop encoder.
> * *Widening overrides*: Add {{readIntegersAsLongs}} and
> {{readIntegersAsDoubles}} that skip the int narrowing step entirely.
> h3. Benchmark Results (GHA, AMD EPYC 7763, JDK 17/21/25)
> ||Type||Speedup||
> |INT32 reads|1.1-1.6x|
> |INT32 skip|1.3-1.8x|
> |INT64 reads|1.8-3.7x|
> |INT64 skip|2.3-4.0x|
> |readIntegersAsLongs (INT32 -> Long)|2.4-2.7x|
> |readIntegersAsDoubles (INT32 -> Double)|2.1-2.4x|
> |readUnsignedLongs|7.3-8.2x|
> PR: https://github.com/apache/spark/pull/55919
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
