Revanth14 opened a new issue, #2360: URL: https://github.com/apache/iceberg-go/issues/2360
### Feature Request / Improvement ## Problem `DataFileStatsFromMeta` in `table/internal/parquet_files.go` creates `nanValueCounts` but never fills it, so data files iceberg-go writes have no `nan_value_counts`. Metrics evaluators can't rule out NaN in those files: after #2355, `NOT(x < 5)` on a float column prunes on `x >= 5 OR is_nan(x)`, and `is_nan` keeps every file iceberg-go wrote, so these filters lose min/max pruning. A strict evaluation can't prove a file is all NaN either. ## Proposal Count NaN values per float and double column while writing, as Java's writers do, and record them in `nan_value_counts`. Parquet column statistics carry no NaN count, so it has to come from the written values. ## Context Raised in #2355. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
