Thanks for writing this up. In addition to being a clear proposal, this is a great explanation of some of the intricacies of timestamps in Beam schemas.
Kenn On Mon, Aug 11, 2025 at 4:34 PM Ahmed Abualsaud via dev <dev@beam.apache.org> wrote: > Hey all, > > A user recently reported issues reading Iceberg timestamps with the Python > SDK. As I investigated, I noticed some gaps in our timestamp story for > IcebergIO (and potentially other IOs). > > I've written a design doc to address these challenges specifically for > IcebergIO. The goal is to establish a more consistent and robust timestamp > strategy that also supports the upcoming nanosecond-precision timestamps in > the Iceberg v3 spec [1]. > > The doc outlines current gaps and proposes a few approaches, including a > preferred one that uses new logical types to ensure accuracy and > flexibility. It also details potential breaking changes and our plan for > managing them. > > Please take a look and share your feedback: > > https://docs.google.com/document/d/19wwp9-4WyE8Ctao0tb1kKCppR4NtvscZ2P2yjokALfQ/edit?usp=sharing > > [1] > https://iceberg.apache.org/spec/#version-3-extended-types-and-capabilities > >
