xndai opened a new pull request, #17557: URL: https://github.com/apache/iceberg/pull/17557
…ount stats Parquet's Statistics#getNumNulls returns -1 when null_count is missing from the stats. But our current metrics logic doesn't handle this -1 value specifically, instead it just adds to the existing count. For a single row group the -1 total was dropped because Metrics only keeps non-negative counts. But say we have row group 0 that has one null value and row group 1 doesn't have null_count stats and return -1, we end up counting total null count as 0, which is wrong. A query engine that relays on total null count being 0, can skip the file completely when evaluating predicate like `WHERE c IS NULL` causing wrong results. Fix by checking the -1 when sum the nulls. Add corresponding tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
