[
https://issues.apache.org/jira/browse/SPARK-57415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57415:
-----------------------------------
Labels: pull-request-available (was: )
> Parquet vectorized reader performance improvements
> --------------------------------------------------
>
> Key: SPARK-57415
> URL: https://issues.apache.org/jira/browse/SPARK-57415
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Umbrella tracking a series of performance improvements to the Parquet
> vectorized reader in Spark SQL. The changes target allocation reduction,
> bulk-read optimizations, and JIT-friendly code patterns across multiple
> encoding paths.
> All subtasks are independent and can be reviewed/merged in any order.
> Together they yield significant throughput gains (1.2x to 9x depending on the
> encoding and data shape) for Parquet reads with no user-facing behavioral
> changes.
> Subtasks:
> - SPARK-56892: Bulk read optimization for Parquet DELTA_BINARY_PACKED
> decoding (up to 9.0x)
> - SPARK-56893: Optimize Parquet dictionary decoding with hasNull fast path
> and per-class updater overrides (1.24x)
> - SPARK-56894: Add vectorized Parquet BYTE_STREAM_SPLIT reader (2.8-4.5x)
> - SPARK-56895: Batch ByteBuffer slice in RLE PACKED decode to reduce
> allocation overhead (2.1-2.4x)
> - SPARK-56896: Add bulk read paths for timestamp/date Parquet vector updaters
> (2.0-2.6x)
> - SPARK-56897: Reduce per-value allocations in DELTA_BYTE_ARRAY Parquet
> decoder (1.1-1.9x)
> - SPARK-56907: Reduce per-value allocation in DELTA_LENGTH_BYTE_ARRAY Parquet
> vectorized reader (1.2-1.4x)
> Common themes:
> - Allocation reduction: Replace per-value ByteBuffer.slice() /
> ByteBuffer.wrap() with bulk reads into reusable buffers
> - Bulk vectorized reads: Replace per-element virtual dispatch with single
> batch calls backed by System.arraycopy
> - JIT-friendly patterns: Per-class method overrides for monomorphic call
> sites; avoiding megamorphic profile pollution
> Benchmarks run on AMD EPYC 7763 with OpenJDK 17/21/25 via GHA workflow.
> GitHub tracking issue: https://github.com/apache/spark/issues/56011
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
