[
https://issues.apache.org/jira/browse/SPARK-56895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56895:
---------------------------------
Description:
Replaces per-group {{in.slice(bitWidth)}} (one {{ByteBuffer}} allocation per 8
values) with a single bulk slice for the entire RLE PACKED run in the Parquet
vectorized reader.
h3. Changes
* *Batch ByteBuffer slice*: Instead of allocating a new {{ByteBuffer}} for each
8-value group in a PACKED run, allocate a single slice covering all groups and
advance a position cursor.
* Eliminates ~128K short-lived ByteBuffer allocations per 1M-value page.
h3. Benchmark Results (GHA, AMD EPYC 7763)
||bitWidth||Speedup (readIntegers)||Speedup (skipIntegers)||
|4|2.0x|2.1x|
|8|2.0x|2.4x|
|12|1.6x|1.6x|
|20|1.4x|1.4x|
PR: https://github.com/apache/spark/pull/55922
Parent issue: https://github.com/apache/spark/issues/56011
> Batch ByteBuffer slice in RLE PACKED decode to reduce allocation overhead
> -------------------------------------------------------------------------
>
> Key: SPARK-56895
> URL: https://issues.apache.org/jira/browse/SPARK-56895
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Replaces per-group {{in.slice(bitWidth)}} (one {{ByteBuffer}} allocation per
> 8 values) with a single bulk slice for the entire RLE PACKED run in the
> Parquet vectorized reader.
> h3. Changes
> * *Batch ByteBuffer slice*: Instead of allocating a new {{ByteBuffer}} for
> each 8-value group in a PACKED run, allocate a single slice covering all
> groups and advance a position cursor.
> * Eliminates ~128K short-lived ByteBuffer allocations per 1M-value page.
> h3. Benchmark Results (GHA, AMD EPYC 7763)
> ||bitWidth||Speedup (readIntegers)||Speedup (skipIntegers)||
> |4|2.0x|2.1x|
> |8|2.0x|2.4x|
> |12|1.6x|1.6x|
> |20|1.4x|1.4x|
> PR: https://github.com/apache/spark/pull/55922
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
