mattfaltyn opened a new issue, #2596:
URL: https://github.com/apache/iceberg-go/issues/2596

   ### Apache Iceberg version
   
   main (development), `7cf76392bbf28bff03bccd1917efebfd5824e4bd`
   
   ### Please describe the bug 🐞
   
   Importing a Parquet file through `Transaction.AddDataFilesByPath` can 
publish incorrect Iceberg file metrics when one row group omits an individual 
statistic. Filters can skip files with matching rows, and a copy-on-write 
delete can remove rows outside its predicate.
   
   #### Reproduction
   
   Use an unpartitioned Iceberg table with one optional `int` column named 
`amount`. Write an ordinary Parquet file with two row groups and import it with 
`AddDataFilesByPath`, then commit. For each case below, omit the indicated 
field from the second row group's footer statistics and leave its pages and 
other statistics intact.
   
   | Row groups | Statistic omitted | Operation | Expected | Actual |
   | --- | --- | --- | --- | --- |
   | `[10]`, `[100, NULL]` | `null_count` | Scan for `amount IS NULL` | One 
null row | Zero rows |
   | `[10]`, `[100, NULL]` | Min/max | Scan for `amount = 100` | One matching 
row | Zero rows |
   | `[100]`, `[100, NULL]` | `null_count` | Copy-on-write delete where `amount 
= 100` | The null row survives | The delete commits and removes all rows |
   
   All three rows are visible in an unfiltered scan before the delete. The same 
cases give the expected results when the footer contains both row groups' 
complete statistics. I reproduced this twice using the local Hadoop catalog on 
Go 1.26.3.
   
   #### Cause
   
   `DataFileStatsFromMeta` sums the null counts it finds and aggregates the 
bounds it finds. A missing statistic in an otherwise present statistics record 
leaves a partial aggregate eligible for publication. For example, the imported 
file can claim a null count of zero or an upper bound of 10.
   
   [Parquet row-group 
statistics](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
 allow individual fields to be absent. [Iceberg file 
metrics](https://iceberg.apache.org/spec/#field-level-metrics-and-statistics) 
describe the entire file. Omit an aggregate when the available chunks cannot 
establish it. A provably all-null chunk contributes no bounds, so complete 
bounds from other chunks remain usable. Partition inference also needs to 
distinguish incomplete bounds from an all-null source.
   
   Related work: #2051 preserves sizes and value counts when a chunk's entire 
statistics record is absent. The cases above keep the statistics record and 
omit individual fields.
   
   I'd like to work on this and will link a focused regression fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to