yihua opened a new issue, #852:
URL: https://github.com/apache/hudi-rs/issues/852
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Description of the bug
`StatisticsContainer::from_parquet_metadata` builds file-level min/max by
merging per-row-group stats with `ColumnStatistics::merge`
(`crates/core/src/statistics/mod.rs`). When one side has a bound and the other
has `None`, `merge` keeps the side that has it:
```rust
(Some(a), None) => Some(a),
(None, Some(b)) => Some(Arc::clone(b)),
```
`None` is ambiguous here. It means "this row group has only nulls" (safe to
skip), but it also means "this row group's min/max is unknown", e.g. the column
chunk has no statistics, or has statistics without min/max. In the second case
the row group can hold any non-null value, so the file-level bound built from
the other row groups is not a bound at all.
`FilePruner` trusts those bounds, so a file can be pruned even though it
contains matching rows. On base-file-only reads (COW, or MOR read-optimized)
where footer stats pruning applies, a filtered read silently returns fewer rows
than it should. This affects every operator the pruner handles (`=`, `<`, `>`,
`IN`, ...), not one in particular.
`test_column_statistics_merge` (case 2) currently asserts the "keep the
`Some` side" behavior.
### Steps To Reproduce
Unit test against `main` (cc5365b5), placed in
`crates/core/src/table/file_pruner.rs`. It builds a two-row-group file: row
group 0 has `id` stats `[50, 100]`, row group 1 has 10 non-null rows and no
column-chunk stats.
```rust
#[cfg(test)]
mod repro_row_group_merge {
use super::*;
use crate::statistics::StatisticsContainer;
use arrow_schema::{DataType, Field};
use parquet::basic::{Repetition, Type as PhysicalType};
use parquet::file::metadata::{
ColumnChunkMetaData, FileMetaData, ParquetMetaData, RowGroupMetaData,
};
use parquet::file::statistics::Statistics;
use parquet::schema::types::{SchemaDescriptor, Type};
use std::sync::Arc;
#[test]
fn repro_row_group_without_stats_is_ignored_and_file_is_pruned() {
let id = Type::primitive_type_builder("id", PhysicalType::INT64)
.with_repetition(Repetition::REQUIRED)
.build()
.unwrap();
let schema = Type::group_type_builder("schema")
.with_fields(vec![Arc::new(id)])
.build()
.unwrap();
let descr = Arc::new(SchemaDescriptor::new(Arc::new(schema)));
// Row group 0: stats say id is in [50, 100].
let rg0 = RowGroupMetaData::builder(descr.clone())
.set_num_rows(10)
.set_column_metadata(vec![
ColumnChunkMetaData::builder(descr.column(0))
.set_num_values(10)
.set_statistics(Statistics::int64(Some(50), Some(100),
None, Some(0), false))
.build()
.unwrap(),
])
.build()
.unwrap();
// Row group 1: 10 non-null rows, no statistics. Its values are
unknown.
let rg1 = RowGroupMetaData::builder(descr.clone())
.set_num_rows(10)
.set_column_metadata(vec![
ColumnChunkMetaData::builder(descr.column(0))
.set_num_values(10)
.build()
.unwrap(),
])
.build()
.unwrap();
let md = ParquetMetaData::new(
FileMetaData::new(1, 20, None, None, descr.clone(), None),
vec![rg0, rg1],
);
let arrow_schema = Schema::new(vec![Field::new("id",
DataType::Int64, false)]);
let stats = StatisticsContainer::from_parquet_metadata(&md,
&arrow_schema);
let filters = vec![Filter::try_from(("id", "=", "5")).unwrap()];
let pruner = FilePruner::new(&filters, &arrow_schema,
&Schema::empty()).unwrap();
assert!(
pruner.should_include(&stats),
"file was pruned for `id = 5` although row group 1 has no stats"
);
}
}
```
```
./build-wrapper.sh cargo test -p hudi-core --lib repro_row_group_merge --
--nocapture
```
### Expected behavior
The file-level min (or max) is `None` whenever any row group that may
contain non-null values has no min (or max). A row group known to be all-null
can still be skipped. With `None` bounds, `FilePruner` keeps the file.
Telling the two cases apart needs the null count and value count per chunk.
`ColumnStatistics` does not carry a null count today, so the conservative
option when that information is missing is to treat a missing bound as unknown.
### Screenshots / Logs
```
file-level id stats: min=Some(PrimitiveArray<Int64>
[
50,
]) max=Some(PrimitiveArray<Int64>
[
100,
])
thread '...repro_row_group_without_stats_is_ignored_and_file_is_pruned'
panicked at crates/core/src/table/file_pruner.rs:696:9:
file was pruned for `id = 5` although row group 1 has no stats
test result: FAILED. 0 passed; 1 failed
```
### Software information
- Operating system: macOS 26 (arm64)
- Rust version: 1.94
- Project version: `main` at cc5365b5
### Additional context
Found while reviewing #800 (IN / NOT IN file pruning), which uses the same
file-level stats. #800 does not introduce the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
