jensholdgaard commented on issue #26490: URL: https://github.com/apache/datafusion/issues/26490#issuecomment-6874919872
> Written by Claude, an AI assistant, on behalf of @jensholdgaard. I ran the tests below and checked the results. Another downstream project is affected: [Ourios](https://github.com/jensholdgaard/ourios), a log store on Parquet + DataFusion. It stores OpenTelemetry double attributes in `Float64` columns, and those values include NaN. With `execution.collect_statistics = true`, a stored NaN reads back as the statistics constant, which changes what we return to users. We keep statistics collection off until a release fixes this. I ran the reproducer below against `56.0.0-rc1` and the current heads of both fix PRs. It writes one row group holding `1.5, NaN, 1.5` with `COPY … STORED AS PARQUET`: | query (`collect_statistics = true`) | 56.0.0-rc1 | #27283 (`504998d`) | #27290 (`e2d5990`) | expected | |---|---|---|---|---| | `count(*) WHERE x < 2.0` | 3 | 2 | 2 | 2 | | `count(*) WHERE isnan(x)` | 0 | 1 | 1 | 1 | | `sum(x)` | 4.5 | NaN | NaN | NaN | | `SELECT x` | 1.5, 1.5, 1.5 | 1.5, NaN, 1.5 | 1.5, NaN, 1.5 | 1.5, NaN, 1.5 | With `collect_statistics = false`, all three versions return the expected values. Both PRs fix our case. <details><summary>Reproducer</summary> ```rust use datafusion::arrow::util::pretty::pretty_format_batches; use datafusion::prelude::*; #[tokio::main] async fn main() -> datafusion::error::Result<()> { let dir = std::env::temp_dir().join("df-nan-repro"); let _ = std::fs::remove_dir_all(&dir); std::fs::create_dir_all(&dir)?; // One row group: 1.5, NaN, 1.5. Parquet min/max leave NaN out, so min == max == 1.5. let file = dir.join("t.parquet"); SessionContext::new() .sql(&format!( "COPY (SELECT column1 AS x FROM (VALUES (1.5::DOUBLE), ('NaN'::DOUBLE), (1.5::DOUBLE))) \ TO '{}' STORED AS PARQUET", file.display() )) .await? .collect() .await?; let queries = [ "SELECT count(*) AS below_2 FROM t WHERE x < 2.0", "SELECT count(*) AS nans FROM t WHERE isnan(x)", "SELECT sum(x) AS total FROM t", "SELECT x FROM t", ]; for collect_statistics in [false, true] { let mut config = SessionConfig::new().with_target_partitions(1); config.options_mut().execution.collect_statistics = collect_statistics; let ctx = SessionContext::new_with_config(config); ctx.register_parquet("t", dir.to_str().unwrap(), ParquetReadOptions::default()).await?; println!("== collect_statistics = {collect_statistics}"); for q in queries { let out = pretty_format_batches(&ctx.sql(q).await?.collect().await?)?.to_string(); let values: Vec<_> = out.lines().skip(3).filter(|l| l.starts_with('|')).map(|l| l.trim_matches(|c| c == '|' || c == ' ')).collect(); println!(" {q:<52} -> {values:?}"); } } Ok(()) } ``` `datafusion = { …, default-features = false, features = ["parquet", "sql", "nested_expressions"] }` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
