hcrosse opened a new pull request, #2675: URL: https://github.com/apache/iceberg-go/pull/2675
## What `AddDataFilesByPath` misinterprets Parquet `TIMESTAMP(MILLIS)` bounds as microseconds and infers incorrect partitions for Iceberg `timestamp` and `timestamptz` columns. Reads can drop matching rows during pruning, fail row filters with a unit mismatch, or miss equality deletes. This PR converts `TIMESTAMP(MILLIS)` to microseconds across both metadata and value reads: - Scales `TIMESTAMP(MILLIS)` `Int64` statistics to microseconds for Iceberg `timestamp` and `timestamptz` fields in `DataFileStatsFromMeta` and `parquetStatsBounds`, fixing manifest bounds, inferred partitions and row group pruning. Bounds that overflow `int64` when scaled are omitted so the file or row group is kept. - Drops dictionary and bloom filter predicates on `TIMESTAMP(MILLIS)` columns in `wrapPqArrowReader.GetRecords` so microsecond filter bytes do not prune matching row groups. - Normalizes `timestamp[ms]` columns to `timestamp[us]` in `wrapPqArrowReader`, including nested struct, list, list view, map and dictionary containers, so row filters and equality deletes use microsecond values. Value overflow returns `iceberg.ErrInvalidArgument`, and files without millisecond columns retain the zero copy path. ## Testing I added unit and end to end tests covering `AddDataFilesByPath` manifest bounds, overflow bound omission and `day(ts)` partition inference, row group statistics, dictionary and bloom pruning, scan row filters with pruning disabled, nested struct timestamps, equality deletes, and overflow error handling under a checked allocator. The new millisecond integration cases fail on `main` and pass with this change, while microsecond controls pass on both. `go test -race ./table/...` and `golangci-lint run ./table/...` pass. Fixes #2674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
