[
https://issues.apache.org/jira/browse/SPARK-56907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-56907:
---------------------------------
Description:
Reduces object allocation in the DELTA_LENGTH_BYTE_ARRAY vectorized Parquet
reader ({{VectorizedDeltaLengthByteArrayReader}}) by applying three targeted
changes:
* *readBinary*: Replace per-value {{in.slice(length)}} (one ByteBuffer
allocation per value) with a single bulk {{in.slice(totalDataLen)}} that reads
the entire batch at once. Individual values are then written to the column
vector via {{putByteArray}} from the shared backing array.
* *skipBinary*: Replace the per-value skip loop (N separate {{in.skip()}}
calls) with a single bulk skip by summing all value lengths upfront.
* *readGeoData*: Remove the {{ByteBuffer.wrap()}} + {{ByteBufferOutputWriter}}
indirection per value and call {{putByteArray}} directly.
h3. Benchmark Results (GHA, AMD EPYC 7763)
||Case||JDK 17||JDK 21||JDK 25||
|readBinary, payloadLen=8|1.14x|1.09x|1.18x|
|readBinary, payloadLen=32|1.10x|0.84x|1.17x|
|skipBinary, payloadLen=8|1.42x|1.69x|1.39x|
|skipBinary, payloadLen=32|1.40x|1.02x|1.34x|
{{readBinary}} speedup is larger for small payloads where allocation cost
dominates. {{skipBinary}} shows consistent improvement from eliminating
per-value stream operations.
PR: https://github.com/apache/spark/pull/55932
Parent issue: https://github.com/apache/spark/issues/56011
> Reduce per-value allocation in DELTA_LENGTH_BYTE_ARRAY Parquet vectorized
> reader
> --------------------------------------------------------------------------------
>
> Key: SPARK-56907
> URL: https://issues.apache.org/jira/browse/SPARK-56907
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Reduces object allocation in the DELTA_LENGTH_BYTE_ARRAY vectorized Parquet
> reader ({{VectorizedDeltaLengthByteArrayReader}}) by applying three targeted
> changes:
> * *readBinary*: Replace per-value {{in.slice(length)}} (one ByteBuffer
> allocation per value) with a single bulk {{in.slice(totalDataLen)}} that
> reads the entire batch at once. Individual values are then written to the
> column vector via {{putByteArray}} from the shared backing array.
> * *skipBinary*: Replace the per-value skip loop (N separate {{in.skip()}}
> calls) with a single bulk skip by summing all value lengths upfront.
> * *readGeoData*: Remove the {{ByteBuffer.wrap()}} +
> {{ByteBufferOutputWriter}} indirection per value and call {{putByteArray}}
> directly.
> h3. Benchmark Results (GHA, AMD EPYC 7763)
> ||Case||JDK 17||JDK 21||JDK 25||
> |readBinary, payloadLen=8|1.14x|1.09x|1.18x|
> |readBinary, payloadLen=32|1.10x|0.84x|1.17x|
> |skipBinary, payloadLen=8|1.42x|1.69x|1.39x|
> |skipBinary, payloadLen=32|1.40x|1.02x|1.34x|
> {{readBinary}} speedup is larger for small payloads where allocation cost
> dominates. {{skipBinary}} shows consistent improvement from eliminating
> per-value stream operations.
> PR: https://github.com/apache/spark/pull/55932
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
