alexschroth opened a new issue, #23352:
URL: https://github.com/apache/datafusion/issues/23352
### Describe the bug
Thanks a lot for all the work on DataFusion! While upgrading from 53 to 54 I
ran into what looks like a regression.
Filtering a `Timestamp(Nanosecond, None)` column whose values span the full
`i64` range (`i64::MIN`..`i64::MAX`) fails during query planning:
**Panics** in **dev** profile:
```
thread 'main' (xxx) panicked at
/path/to/datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23:
attempt to add with overflow
stack backtrace:
...
3:
datafusion_expr_common::interval_arithmetic::Interval::cardinality::{{closure}}
at
datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23
4: core::option::Option<T>::map
at rust/library/core/src/option.rs:1165:29
5: datafusion_expr_common::interval_arithmetic::Interval::cardinality
at
datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:10
6: datafusion_expr_common::interval_arithmetic::cardinality_ratio
at
datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:1723:59
7: datafusion_physical_expr::analysis::calculate_selectivity
at datafusion-physical-expr-54.0.0/src/analysis.rs:331:24
8: datafusion_physical_expr::analysis::shrink_boundaries
at datafusion-physical-expr-54.0.0/src/analysis.rs:253:23
9: datafusion_physical_expr::analysis::analyze
at datafusion-physical-expr-54.0.0/src/analysis.rs:217:17
10: datafusion_physical_plan::filter::FilterExec::statistics_helper
at datafusion-physical-plan-54.0.0/src/filter.rs:365:32
11: datafusion_physical_plan::filter::FilterExec::compute_properties
at datafusion-physical-plan-54.0.0/src/filter.rs:401:21
12: datafusion_physical_plan::filter::FilterExecBuilder::build
at datafusion-physical-plan-54.0.0/src/filter.rs:201:21
```
Returns an **error** in **release** profile:
```
Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity):
Selectivity is out of limit: inf")
```
I have also seen the same error with `NaN` instead of `inf` in the wild:
```
Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity):
Selectivity is out of limit: NaN")
```
but unfortunately I have not been able to put together a minimal reproducer
for the `NaN` variant.
This worked fine in DataFusion 53.
### To Reproduce
`Cargo.toml`:
```toml
[dependencies]
datafusion = { version = "54", default-features = false, features =
["parquet"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```
`src/main.rs`:
```rust
use datafusion::arrow::array::TimestampNanosecondArray;
use datafusion::arrow::datatypes::{DataType, Field, Schema, TimeUnit};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;
use datafusion::scalar::ScalarValue;
use std::error::Error;
use std::fs::File;
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let parquet_file = "test.parquet";
let schema = Arc::new(Schema::new(vec![Field::new(
"ts",
DataType::Timestamp(TimeUnit::Nanosecond, None),
true,
)]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(TimestampNanosecondArray::from(vec![
i64::MIN,
i64::MAX,
]))],
)?;
let mut writer = ArrowWriter::try_new(File::create(parquet_file)?,
schema, None)?;
writer.write(&batch)?;
writer.close()?;
let ctx = SessionContext::new();
ctx.register_parquet("t", parquet_file, Default::default())
.await?;
let batches = ctx
.table("t")
.await?
.filter(col("ts").lt_eq(lit(ScalarValue::TimestampNanosecond(
Some(1_000_000_000_000_000_000),
None,
))))?
.collect()
.await?;
println!(
"collected {} batch(es), {} row(s)",
batches.len(),
batches.iter().map(|b| b.num_rows()).sum::<usize>()
);
Ok(())
}
```
Run it:
```
$ cargo run
thread 'main' panicked at
datafusion-expr-common-54.0.0/src/interval_arithmetic.rs:947:23:
attempt to add with overflow
$ cargo run --release
Error: Internal("Assertion failed: (0.0..=1.0).contains(&selectivity):
Selectivity is out of limit: inf")
```
### Expected behavior
The query plans and runs successfully and returns the matching rows, as it
does on DataFusion 53:
```
collected 1 batch(es), 1 row(s)
```
### Additional context
- Reproduced identically on macOS 26.5.1 (arm64) and Ubuntu 24.04 (x86_64)
- Versions: datafusion 54.0.0, arrow 58.3.0, rustc 1.96.1 (2026-06-26), Rust
edition 2024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
