[
https://issues.apache.org/jira/browse/SPARK-56894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-56894.
------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 55921
[https://github.com/apache/spark/pull/55921]
> Add vectorized Parquet BYTE_STREAM_SPLIT reader
> -----------------------------------------------
>
> Key: SPARK-56894
> URL: https://issues.apache.org/jira/browse/SPARK-56894
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Adds a new {{VectorizedByteStreamSplitValuesReader}} that decodes
> BYTE_STREAM_SPLIT-encoded Parquet pages (FLOAT, DOUBLE, INT32, INT64,
> FIXED_LEN_BYTE_ARRAY) using batch byte-gathering instead of falling back to
> parquet-mr per-value reads.
> h3. Changes
> * *Batch byte-gathering decode*: Reads all stream bytes in bulk and
> reassembles values by gathering the i-th byte from each of N interleaved
> streams.
> * *Widening overrides*: {{readIntegersAsLongs}}, {{readFloatsAsDoubles}}
> avoid intermediate narrowing.
> * *FLBA batch allocation reduction*: Avoids per-value ByteBuffer allocation
> for FIXED_LEN_BYTE_ARRAY.
> * *FixedLenByteArrayUpdater routing*: Routes through the batch path.
> h3. Benchmark Results (GHA, AMD EPYC 7763)
> ||Type||JDK 17||JDK 21||JDK 25||
> |INT32|4.5x|3.8x|4.2x|
> |INT64|2.8x|2.0x|1.8x|
> |FLOAT|4.3x|3.6x|4.3x|
> |DOUBLE|2.7x|2.0x|1.8x|
> |readIntegersAsLongs|3.8x|3.0x|3.5x|
> |readFloatsAsDoubles|3.8x|3.2x|3.9x|
> |FLBA(12) readBinary|1.7x|1.6x|1.5x|
> PR: https://github.com/apache/spark/pull/55921
> Parent issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
