[
https://issues.apache.org/jira/browse/SPARK-58735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eyck Troschke updated SPARK-58735:
----------------------------------
Description:
When applying `size()` to a nested array column stored in Parquet/Delta, Spark
reads the *entire* array column including all child struct fields, even though
only array offsets / repetition levels are required to compute the array
length. This results in significantly higher I/O compared to semantically
equivalent queries using `explode()` or accessing smaller sub‑arrays.
This behavior occurs in both Apache Spark and Databricks runtimes. It appears
to be caused by conservative Catalyst planning for nested arrays, which
prevents column pruning and page skipping.
*Example:*
Table size: ~100GB, ~18B rows (after explode)
Column: `events` (ARRAY<STRUCT<...>>)
Sub‑column: `events.flag_a` (Boolean)
```
select sum(size(events)); -- 49s, ~100GB read
select sum(size(events.flag_a)); -- 8s, ~6GB read
select count(*) from t lateral view explode(events); -- 9s, ~10GB read
select count(*) from t lateral view explode(events.flag_a); – 8s, ~6GB read
```
Although these queries are semantically equivalent, `size(array)` triggers a
full scan of all nested array fields, while `explode(array)` allows column
pruning.
*Expected Behavior:*
`size(array)` should only require reading array offsets / repetition levels and
should not force full read of all nested struct fields.
*Actual Behavior:*
Spark reads all nested array child columns, causing large and unnecessary I/O.
*Impact:*
Severe performance degradation for large nested arrays in Parquet/Delta tables.
was:
When applying `size()` to a nested array column stored in Parquet/Delta, Spark
reads the *entire* array column including all child struct fields, even though
only array offsets / repetition levels are required to compute the array
length. This results in significantly higher I/O compared to semantically
equivalent queries using `explode()` or accessing smaller sub‑arrays.
This behavior occurs in both Apache Spark and Databricks runtimes. It appears
to be caused by conservative Catalyst planning for nested arrays, which
prevents column pruning and page skipping.
*Example:*
Table size: ~100GB, ~18B rows
Column: `events` (ARRAY<STRUCT<...>>)
Sub‑column: `events.flag_a` (small ARRAY)
```
select sum(size(events)); -- 49s, ~100GB read
select sum(size(events.flag_a)); -- 8s, ~6GB read
select count(*) from t lateral view explode(events); -- 9s, ~10GB read
select count(*) from t lateral view explode(events.flag_a); -- 8s, ~6GB read
```
Although these queries are semantically equivalent, `size(array)` triggers a
full scan of all nested array fields, while `explode(array)` allows column
pruning.
*Expected Behavior:*
`size(array)` should only require reading array offsets / repetition levels and
should not force full read of all nested struct fields.
*Actual Behavior:*
Spark reads all nested array child columns, causing large and unnecessary I/O.
*Impact:*
Severe performance degradation for large nested arrays in Parquet/Delta tables.
> "size(array)" forces full read of nested array columns and prevents column
> pruning for Parquet/Delta arrays
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58735
> URL: https://issues.apache.org/jira/browse/SPARK-58735
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
> Affects Versions: 4.1.2
> Environment: Apache Spark 4.1.2
> Databricks Runtime (Photon)
> Reporter: Eyck Troschke
> Priority: Major
> Attachments: test_size_vs_explode.py
>
>
> When applying `size()` to a nested array column stored in Parquet/Delta,
> Spark reads the *entire* array column including all child struct fields, even
> though only array offsets / repetition levels are required to compute the
> array length. This results in significantly higher I/O compared to
> semantically equivalent queries using `explode()` or accessing smaller
> sub‑arrays.
> This behavior occurs in both Apache Spark and Databricks runtimes. It appears
> to be caused by conservative Catalyst planning for nested arrays, which
> prevents column pruning and page skipping.
> *Example:*
> Table size: ~100GB, ~18B rows (after explode)
> Column: `events` (ARRAY<STRUCT<...>>)
> Sub‑column: `events.flag_a` (Boolean)
> ```
> select sum(size(events)); -- 49s, ~100GB read
> select sum(size(events.flag_a)); -- 8s, ~6GB read
> select count(*) from t lateral view explode(events); -- 9s, ~10GB
> read
> select count(*) from t lateral view explode(events.flag_a); – 8s, ~6GB read
> ```
> Although these queries are semantically equivalent, `size(array)` triggers a
> full scan of all nested array fields, while `explode(array)` allows column
> pruning.
> *Expected Behavior:*
> `size(array)` should only require reading array offsets / repetition levels
> and should not force full read of all nested struct fields.
> *Actual Behavior:*
> Spark reads all nested array child columns, causing large and unnecessary I/O.
> *Impact:*
> Severe performance degradation for large nested arrays in Parquet/Delta
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
