hcrosse opened a new pull request, #2675:
URL: https://github.com/apache/iceberg-go/pull/2675

   ## What
   
   `AddDataFilesByPath` misinterprets Parquet `TIMESTAMP(MILLIS)` bounds as 
microseconds and infers incorrect partitions for Iceberg `timestamp` and 
`timestamptz` columns. Reads can drop matching rows during pruning, fail row 
filters with a unit mismatch, or miss equality deletes.
   
   This PR converts `TIMESTAMP(MILLIS)` to microseconds across both metadata 
and value reads:
   
   - Scales `TIMESTAMP(MILLIS)` `Int64` statistics to microseconds for Iceberg 
`timestamp` and `timestamptz` fields in `DataFileStatsFromMeta` and 
`parquetStatsBounds`, fixing manifest bounds, inferred partitions and row group 
pruning. Bounds that overflow `int64` when scaled are omitted so the file or 
row group is kept.
   - Drops dictionary and bloom filter predicates on `TIMESTAMP(MILLIS)` 
columns in `wrapPqArrowReader.GetRecords` so microsecond filter bytes do not 
prune matching row groups.
   - Normalizes `timestamp[ms]` columns to `timestamp[us]` in 
`wrapPqArrowReader`, including nested struct, list, list view, map and 
dictionary containers, so row filters and equality deletes use microsecond 
values. Value overflow returns `iceberg.ErrInvalidArgument`, and files without 
millisecond columns retain the zero copy path.
   
   ## Testing
   
   I added unit and end to end tests covering `AddDataFilesByPath` manifest 
bounds, overflow bound omission and `day(ts)` partition inference, row group 
statistics, dictionary and bloom pruning, scan row filters with pruning 
disabled, nested struct timestamps, equality deletes, and overflow error 
handling under a checked allocator. The new millisecond integration cases fail 
on `main` and pass with this change, while microsecond controls pass on both.
   
   `go test -race ./table/...` and `golangci-lint run ./table/...` pass.
   
   Fixes #2674
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to