Revanth14 opened a new pull request, #2420: URL: https://github.com/apache/iceberg-go/pull/2420
## Problem `DataFileStatsFromMeta` creates `nanValueCounts` but never fills it, so data files iceberg-go writes have no `nan_value_counts` (#2360). Metrics evaluators then can't rule NaN out of those files: an inclusive `is_nan` keeps every one, so `NOT(x < 5)`, which prunes on `x >= 5 OR is_nan(x)` after #2355, loses min/max pruning, and a strict evaluation can't prove a file is all NaN. ## Fix Parquet column statistics carry no NaN count, so `ParquetFileWriter` counts NaN values in each batch as it writes them, the way it already collects geo bounds, and records the counts at `Close`. Which fields get a count follows Java's `MetricsUtil.createNanValueCounts`: float and double leaves, including those nested in structs, but not inside a list or map, and not in columns whose metrics mode is `none`. A column with no NaN records 0, which is what lets an evaluator rule NaN out. A value under a null struct row isn't counted. Bounds are unchanged; Parquet already leaves NaN out of float min/max. Files added by path (`AddDataFilesByPath`) still get no NaN counts, since only the footer is read; Java's footer metrics have none either. ## Testing `TestWriteDataFileRecordsNaNValueCounts` writes two batches through `WriteDataFile` and checks the exact `nan_value_counts`. It covers: - a double column with NaN in both batches and a NULL row - a float column in counts mode - a double inside a struct with a NaN under a null struct row - a list of doubles, which is excluded - a column with metrics mode `none`, which is excluded - a column with no NaN, which records 0 Breaking each rule separately (nulls under a struct, applying the counts, list exclusion, mode `none`, zero counts) fails the test. ## Context Closes #2360. Follow-up from #2355, which should merge first: with complete NaN counts, `main`'s float32 out-of-range fold to always-true would let the strict whole-file delete drop an all-NaN file that iceberg-go wrote; #2355 replaces that fold. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
