jensholdgaard opened a new issue, #27415:
URL: https://github.com/apache/datafusion/issues/27415
> [!NOTE]
> This report was investigated and written with AI assistance (Claude Code),
posted with the account owner's review and consent.
## Describe the bug
Follow-up to #24769 / #24770, tested on the `56.0.0-rc1` tag (`cce6374`).
With `datafusion.execution.collect_statistics = true`, a row group that its
statistics prove cannot match is still scanned, if the predicate has a term
that isn't usable for pruning: here `array_element` over a list column. The
same query prunes the row group when `collect_statistics = false`. Without the
`array_element` term, rc1 prunes it with statistics on as well, which is what
#24770 fixed.
| predicate (b.parquet can match neither) | stats off | stats on, 55.2.0 |
stats on, 56.0.0-rc1 |
|---|---|---|---|
| `body = 'x' OR (body IS NULL AND tid = 1)` | pruned | **scanned** | pruned
(#24770) |
| `body = 'x' OR (body IS NULL AND tid = 1 AND array_element(seps, 1) = '')`
| pruned | **scanned** | **scanned** |
The `pruning_predicate` printed in `EXPLAIN ANALYZE` is identical with
statistics on and off, and the same for both predicates, because the
`array_element` term is dropped from it:
```
pruning_predicate=body_null_count@2 != row_count@3 AND body_min@0 <= x AND x
<= body_max@1 OR body_null_count@2 > 0 AND tid_null_count@6 != row_count@3 AND
tid_min@4 <= 1 AND 1 <= tid_max@5
```
So the difference seems to come from the per-file rewrite that runs once
file statistics are present, the path #24770 changed. It looks like that
rewrite gives up when the folded predicate still holds a term it can't evaluate
from statistics, instead of keeping the unknown term as `true` the way the
pruning predicate does.
Results are correct; only pruning is lost. In our case (a log store built on
DataFusion, where these predicates come from a templated body-equality rewrite)
it keeps `collect_statistics` off, since otherwise every selective query scans
every row group of every file.
This is not a regression from 55: 55.2.0 scans in both cases. So I don't
think it blocks the 56.0.0 vote.
## To Reproduce
`Cargo.toml`:
```toml
[package]
name = "df56-repro"
version = "0.1.0"
edition = "2024"
publish = false
[dependencies]
datafusion = { git = "https://github.com/apache/datafusion";, tag =
"56.0.0-rc1", default-features = false, features = ["parquet", "sql",
"nested_expressions"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```
`src/main.rs`:
```rust
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
let root = std::env::temp_dir().join("df-pruning-repro");
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(&root)?;
// Two single-row-group files. `body` is entirely NULL in both; `tid` is
1 in a.parquet, 9 in b.parquet.
let setup = SessionContext::new();
for (file, tid) in [("a", 1), ("b", 9)] {
let sql = format!(
"COPY (SELECT CAST(NULL AS VARCHAR) AS body, CAST({tid} AS
BIGINT) AS tid, make_array('', '') AS seps) \
TO '{}' STORED AS PARQUET",
root.join(format!("{file}.parquet")).display()
);
setup.sql(&sql).await?.collect().await?;
}
// b.parquet cannot match either predicate: body is all NULL and tid is
9.
let predicates = [
"body = 'x' OR (body IS NULL AND tid = 1)",
"body = 'x' OR (body IS NULL AND tid = 1 AND array_element(seps, 1)
= '')",
];
for collect_statistics in [false, true] {
let mut config = SessionConfig::new().with_target_partitions(1);
config.options_mut().execution.collect_statistics =
collect_statistics;
let ctx = SessionContext::new_with_config(config);
ctx.register_parquet("t", root.to_str().unwrap(),
ParquetReadOptions::default()).await?;
for predicate in predicates {
let batches = ctx
.sql(&format!("EXPLAIN ANALYZE SELECT count(*) FROM t WHERE
{predicate}"))
.await?
.collect()
.await?;
let plan =
datafusion::arrow::util::pretty::pretty_format_batches(&batches)?.to_string();
let pruning = plan
.split("row_groups_pruned_statistics=")
.nth(1)
.and_then(|rest| rest.split(',').next())
.unwrap_or("?");
println!("collect_statistics={collect_statistics:<5}
{predicate}\n row_groups_pruned_statistics={pruning}");
}
}
Ok(())
}
```
Output on `56.0.0-rc1`:
```
collect_statistics=false body = 'x' OR (body IS NULL AND tid = 1)
row_groups_pruned_statistics=2 total → 1 matched
collect_statistics=false body = 'x' OR (body IS NULL AND tid = 1 AND
array_element(seps, 1) = '')
row_groups_pruned_statistics=2 total → 1 matched
collect_statistics=true body = 'x' OR (body IS NULL AND tid = 1)
row_groups_pruned_statistics=2 total → 1 matched
collect_statistics=true body = 'x' OR (body IS NULL AND tid = 1 AND
array_element(seps, 1) = '')
row_groups_pruned_statistics=2 total → 2 matched
```
On `55.2.0` (`datafusion = { version = "=55.2.0", … }`), the third line also
reads `2 total → 2 matched`; the other lines are the same.
## Expected behavior
With `collect_statistics = true`, b.parquet's row group is pruned for both
predicates, as it is with `collect_statistics = false`. A term that can't be
evaluated from statistics should only make its own branch unknown. It shouldn't
stop the rest of the predicate from pruning.
## Additional context
- Seen first through Ourios's pruning acceptance tests while testing the rc.
Our predicates carry `array_element(separators, n) = '…'` terms beside the
`template_id` and `body` equality.
- `body = 'x' OR (tid = 1 AND array_element(seps, 1) = '')`, without the
`body IS NULL` conjunct, isn't pruned with statistics off either. That looks
like a separate, older limitation, and I've left it out of this report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
