jensholdgaard commented on issue #26490:
URL: https://github.com/apache/datafusion/issues/26490#issuecomment-6874919872

   > Written by Claude, an AI assistant, on behalf of @jensholdgaard. I ran the 
tests below and checked the results.
   
   Another downstream project is affected: 
[Ourios](https://github.com/jensholdgaard/ourios), a log
   store on Parquet + DataFusion. It stores OpenTelemetry double attributes in 
`Float64` columns,
   and those values include NaN. With `execution.collect_statistics = true`, a 
stored NaN reads back
   as the statistics constant, which changes what we return to users. We keep 
statistics
   collection off until a release fixes this.
   
   I ran the reproducer below against `56.0.0-rc1` and the current heads of 
both fix PRs. It writes
   one row group holding `1.5, NaN, 1.5` with `COPY … STORED AS PARQUET`:
   
   | query (`collect_statistics = true`) | 56.0.0-rc1 | #27283 (`504998d`) | 
#27290 (`e2d5990`) | expected |
   |---|---|---|---|---|
   | `count(*) WHERE x < 2.0` | 3 | 2 | 2 | 2 |
   | `count(*) WHERE isnan(x)` | 0 | 1 | 1 | 1 |
   | `sum(x)` | 4.5 | NaN | NaN | NaN |
   | `SELECT x` | 1.5, 1.5, 1.5 | 1.5, NaN, 1.5 | 1.5, NaN, 1.5 | 1.5, NaN, 1.5 
|
   
   With `collect_statistics = false`, all three versions return the expected 
values. Both PRs
   fix our case.
   
   <details><summary>Reproducer</summary>
   
   ```rust
   use datafusion::arrow::util::pretty::pretty_format_batches;
   use datafusion::prelude::*;
   
   #[tokio::main]
   async fn main() -> datafusion::error::Result<()> {
       let dir = std::env::temp_dir().join("df-nan-repro");
       let _ = std::fs::remove_dir_all(&dir);
       std::fs::create_dir_all(&dir)?;
       // One row group: 1.5, NaN, 1.5. Parquet min/max leave NaN out, so min 
== max == 1.5.
       let file = dir.join("t.parquet");
       SessionContext::new()
           .sql(&format!(
               "COPY (SELECT column1 AS x FROM (VALUES (1.5::DOUBLE), 
('NaN'::DOUBLE), (1.5::DOUBLE))) \
                TO '{}' STORED AS PARQUET",
               file.display()
           ))
           .await?
           .collect()
           .await?;
       let queries = [
           "SELECT count(*) AS below_2 FROM t WHERE x < 2.0",
           "SELECT count(*) AS nans FROM t WHERE isnan(x)",
           "SELECT sum(x) AS total FROM t",
           "SELECT x FROM t",
       ];
       for collect_statistics in [false, true] {
           let mut config = SessionConfig::new().with_target_partitions(1);
           config.options_mut().execution.collect_statistics = 
collect_statistics;
           let ctx = SessionContext::new_with_config(config);
           ctx.register_parquet("t", dir.to_str().unwrap(), 
ParquetReadOptions::default()).await?;
           println!("== collect_statistics = {collect_statistics}");
           for q in queries {
               let out = 
pretty_format_batches(&ctx.sql(q).await?.collect().await?)?.to_string();
               let values: Vec<_> = out.lines().skip(3).filter(|l| 
l.starts_with('|')).map(|l| l.trim_matches(|c| c == '|' || c == ' ')).collect();
               println!("  {q:<52} -> {values:?}");
           }
       }
       Ok(())
   }
   ```
   
   `datafusion = { …, default-features = false, features = ["parquet", "sql", 
"nested_expressions"] }`
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to