[
https://issues.apache.org/jira/browse/SPARK-57415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated SPARK-57415:
---------------------------------
Description:
Umbrella tracking a series of performance improvements to the Parquet
vectorized reader in Spark SQL. The changes target allocation reduction,
bulk-read optimizations, and JIT-friendly code patterns across multiple
encoding paths.
All subtasks are independent and can be reviewed/merged in any order. Together
they yield significant throughput gains (1.2x to 9x depending on the encoding
and data shape) for Parquet reads with no user-facing behavioral changes.
Common themes:
- Allocation reduction: Replace per-value ByteBuffer.slice() /
ByteBuffer.wrap() with bulk reads into reusable buffers
- Bulk vectorized reads: Replace per-element virtual dispatch with single
batch calls backed by System.arraycopy
- JIT-friendly patterns: Per-class method overrides for monomorphic call
sites; avoiding megamorphic profile pollution
Benchmarks with OpenJDK 17/21/25 via GHA workflow.
GitHub tracking issue: [https://github.com/apache/spark/issues/56011]
was:
Umbrella tracking a series of performance improvements to the Parquet
vectorized reader in Spark SQL. The changes target allocation reduction,
bulk-read optimizations, and JIT-friendly code patterns across multiple
encoding paths.
All subtasks are independent and can be reviewed/merged in any order. Together
they yield significant throughput gains (1.2x to 9x depending on the encoding
and data shape) for Parquet reads with no user-facing behavioral changes.
Common themes:
- Allocation reduction: Replace per-value ByteBuffer.slice() /
ByteBuffer.wrap() with bulk reads into reusable buffers
- Bulk vectorized reads: Replace per-element virtual dispatch with single batch
calls backed by System.arraycopy
- JIT-friendly patterns: Per-class method overrides for monomorphic call sites;
avoiding megamorphic profile pollution
Benchmarks run on AMD EPYC 7763 with OpenJDK 17/21/25 via GHA workflow.
GitHub tracking issue: https://github.com/apache/spark/issues/56011
> Parquet vectorized reader performance improvements
> --------------------------------------------------
>
> Key: SPARK-57415
> URL: https://issues.apache.org/jira/browse/SPARK-57415
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
>
> Umbrella tracking a series of performance improvements to the Parquet
> vectorized reader in Spark SQL. The changes target allocation reduction,
> bulk-read optimizations, and JIT-friendly code patterns across multiple
> encoding paths.
> All subtasks are independent and can be reviewed/merged in any order.
> Together they yield significant throughput gains (1.2x to 9x depending on the
> encoding and data shape) for Parquet reads with no user-facing behavioral
> changes.
> Common themes:
> - Allocation reduction: Replace per-value ByteBuffer.slice() /
> ByteBuffer.wrap() with bulk reads into reusable buffers
> - Bulk vectorized reads: Replace per-element virtual dispatch with single
> batch calls backed by System.arraycopy
> - JIT-friendly patterns: Per-class method overrides for monomorphic call
> sites; avoiding megamorphic profile pollution
> Benchmarks with OpenJDK 17/21/25 via GHA workflow.
> GitHub tracking issue: [https://github.com/apache/spark/issues/56011]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
