xndai opened a new issue, #17561: URL: https://github.com/apache/iceberg/issues/17561
### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 When an optional struct is null, OptionWriter writes a null directly to every leaf column it contains, so the writers for those columns never see the value and cannot count it. OptionWriter dropped its own null count in that case, with a comment saying nested null stats were not used. They are used: the counts it returns become DataFile.nullValueCounts. Float and double are the only types whose writers report metrics, and ParquetMetrics prefers writer metrics over footer statistics, so the correct footer count was never used. A float or double under a nullable struct was reported as having 0 nulls even when the struct was null for some rows. The incorrect counting of nulls could affect query engines that relay on this stats for optimization. For example, they could simply skip the file with null_count == 0 for predicate WHERE c.f_id IS NULL and produces wrong result. Fix by PR: https://github.com/apache/iceberg/pull/17560 ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
