mattfaltyn opened a new issue, #2596: URL: https://github.com/apache/iceberg-go/issues/2596
### Apache Iceberg version main (development), `7cf76392bbf28bff03bccd1917efebfd5824e4bd` ### Please describe the bug 🐞 Importing a Parquet file through `Transaction.AddDataFilesByPath` can publish incorrect Iceberg file metrics when one row group omits an individual statistic. Filters can skip files with matching rows, and a copy-on-write delete can remove rows outside its predicate. #### Reproduction Use an unpartitioned Iceberg table with one optional `int` column named `amount`. Write an ordinary Parquet file with two row groups and import it with `AddDataFilesByPath`, then commit. For each case below, omit the indicated field from the second row group's footer statistics and leave its pages and other statistics intact. | Row groups | Statistic omitted | Operation | Expected | Actual | | --- | --- | --- | --- | --- | | `[10]`, `[100, NULL]` | `null_count` | Scan for `amount IS NULL` | One null row | Zero rows | | `[10]`, `[100, NULL]` | Min/max | Scan for `amount = 100` | One matching row | Zero rows | | `[100]`, `[100, NULL]` | `null_count` | Copy-on-write delete where `amount = 100` | The null row survives | The delete commits and removes all rows | All three rows are visible in an unfiltered scan before the delete. The same cases give the expected results when the footer contains both row groups' complete statistics. I reproduced this twice using the local Hadoop catalog on Go 1.26.3. #### Cause `DataFileStatsFromMeta` sums the null counts it finds and aggregates the bounds it finds. A missing statistic in an otherwise present statistics record leaves a partial aggregate eligible for publication. For example, the imported file can claim a null count of zero or an upper bound of 10. [Parquet row-group statistics](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) allow individual fields to be absent. [Iceberg file metrics](https://iceberg.apache.org/spec/#field-level-metrics-and-statistics) describe the entire file. Omit an aggregate when the available chunks cannot establish it. A provably all-null chunk contributes no bounds, so complete bounds from other chunks remain usable. Partition inference also needs to distinguish incomplete bounds from an all-null source. Related work: #2051 preserves sizes and value counts when a chunk's entire statistics record is absent. The cases above keep the statistics record and omit individual fields. I'd like to work on this and will link a focused regression fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
