[
https://issues.apache.org/jira/browse/SPARK-56897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56897:
---------------------------------
Description:
Reduces per-value heap allocations in {{VectorizedDeltaByteArrayReader}} (the
Parquet DELTA_BYTE_ARRAY vectorized decoder) by replacing {{ByteBuffer}}-based
state tracking with a reusable {{byte[]}} buffer.
h3. Changes
* Replace {{ByteBuffer previous}} with {{byte[] prevBuf}} + {{int prevLen}}:
the old code materialized each decoded value as a {{ByteBuffer}} obtained from
{{arrayData.getByteBuffer()}} (which allocates ~48 bytes per value via
{{ByteBuffer.wrap()}}). The new approach uses a reusable {{byte[]}} buffer
where prefix bytes from the previous iteration are already in place.
* Add {{getSuffixInto()}} to {{VectorizedDeltaLengthByteArrayReader}}: reads
suffix bytes directly into a caller-supplied {{byte[]}} via
{{InputStream.read()}}, avoiding the {{ByteBuffer}} allocation.
* Rewrite {{skipBinary()}} to use {{prevBuf}} directly: eliminates column
vector reset/swap overhead.
* Remove {{tempBinaryValVector}} field: no longer needed.
h3. Benchmark Results (GHA, AMD EPYC 7763)
*skipBinary* (primary improvement):
||Case||JDK 17||JDK 21||JDK 25||
|no overlap|1.35x|1.39x|1.49x|
|half overlap|1.62x|1.62x|1.72x|
|full overlap|1.61x|1.64x|1.74x|
*readBinary* (neutral to modest improvement):
||Case||JDK 17||JDK 21||JDK 25||
|no overlap|0.92x|0.97x|1.06x|
|half overlap|1.07x|1.11x|1.22x|
PR: https://github.com/apache/spark/pull/55924
Parent issue: https://github.com/apache/spark/issues/56011
> Reduce per-value allocations in DELTA_BYTE_ARRAY Parquet decoder
> ----------------------------------------------------------------
>
> Key: SPARK-56897
> URL: https://issues.apache.org/jira/browse/SPARK-56897
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Reduces per-value heap allocations in {{VectorizedDeltaByteArrayReader}} (the
> Parquet DELTA_BYTE_ARRAY vectorized decoder) by replacing
> {{ByteBuffer}}-based state tracking with a reusable {{byte[]}} buffer.
> h3. Changes
> * Replace {{ByteBuffer previous}} with {{byte[] prevBuf}} + {{int prevLen}}:
> the old code materialized each decoded value as a {{ByteBuffer}} obtained
> from {{arrayData.getByteBuffer()}} (which allocates ~48 bytes per value via
> {{ByteBuffer.wrap()}}). The new approach uses a reusable {{byte[]}} buffer
> where prefix bytes from the previous iteration are already in place.
> * Add {{getSuffixInto()}} to {{VectorizedDeltaLengthByteArrayReader}}: reads
> suffix bytes directly into a caller-supplied {{byte[]}} via
> {{InputStream.read()}}, avoiding the {{ByteBuffer}} allocation.
> * Rewrite {{skipBinary()}} to use {{prevBuf}} directly: eliminates column
> vector reset/swap overhead.
> * Remove {{tempBinaryValVector}} field: no longer needed.
> h3. Benchmark Results (GHA, AMD EPYC 7763)
> *skipBinary* (primary improvement):
> ||Case||JDK 17||JDK 21||JDK 25||
> |no overlap|1.35x|1.39x|1.49x|
> |half overlap|1.62x|1.62x|1.72x|
> |full overlap|1.61x|1.64x|1.74x|
> *readBinary* (neutral to modest improvement):
> ||Case||JDK 17||JDK 21||JDK 25||
> |no overlap|0.92x|0.97x|1.06x|
> |half overlap|1.07x|1.11x|1.22x|
> PR: https://github.com/apache/spark/pull/55924
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
