aIbrahiim commented on issue #34809: URL: https://github.com/apache/beam/issues/34809#issuecomment-5156471636
Hi @ahmedabu98 while looking at the failing logs of this issue, it seems related to #39344. What I see The Dataflow managed Iceberg ITs started failing. Direct runner Iceberg ITs still look fine and the failure shows up on Dataflow. In testWrite, after the job finishes, the table schema does not match the Beam schema: > expected: datetime_tz as beam:logical_type:timestamp:v1 > actual: datetime_tz as a plain ROW with seconds / subseconds That suggests the Iceberg table was created with a struct, not timestamptz. In testRead / related jobs, workers fail with: > ClassCastException: org.joda.time.Instant cannot be cast to java.time.Instant > at Timestamp.toBaseType Probably why this happening is because of : - On Dataflow/portable pipelines, Timestamp can show up as UnknownLogicalType (a PassThroughLogicalType) - In IcebergUtils.beamFieldTypeToIcebergFieldType, PassThrough is handled before the Timestamp URN. If that path is taken, the Row base type is used, which becomes an Iceberg struct instead of timestamptz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
