rangareddy commented on issue #581:
URL:
https://github.com/apache/incubator-xtable/issues/581#issuecomment-5056397001
Root cause: this is a mismatch between the **declared schema** and the
**physical parquet data** in the source Hudi table, not an XTable conversion
bug.
- The Hudi table's Avro schema declares `last_modified_time` as a UTC
timestamp logical type (`timestamp-micros`/`timestamp-millis`), so XTable maps
it to Iceberg `timestamp with time zone`:
- `AvroSchemaConverter` maps Avro `long` +
`TimestampMillis`/`TimestampMicros` → internal `TIMESTAMP`.
- `IcebergSchemaExtractor` maps internal `TIMESTAMP` →
`Types.TimestampType.withZone()` (i.e. `timestamp(6) with time zone`).
- But the physical parquet column is a bare `INT64` with no timestamp
logical annotation. The stored value `1732162935` is epoch **seconds** (Nov
2024), not the micros (`~1.7e18`) or millis (`~1.7e12`) a real timestamp column
would hold — so the data was written as a plain bigint while the schema labels
it a timestamp.
When Athena/Iceberg reads the file, the Iceberg schema says `timestamptz`
but the parquet physical type is bare `INT64` → `ICEBERG_BAD_DATA ...
incompatible`. XTable only translates metadata and never rewrites data files,
so it cannot reconcile a source table whose schema and data already disagree.
This is effectively the UTC-timestamp sibling of #672 ("Hudi 0.x writes
INT64 in parquet for `LocalTimestampMillis`"). XTable already works around the
*local*-timestamp variant by mapping `local-timestamp-*` to `long` (bigint)
specifically to avoid this Iceberg-read failure.
### Suggested resolution
1. If `last_modified_time` actually holds epoch-seconds, declare it as
`bigint`/`long` in the Hudi table (not `timestamp`); XTable will then emit
Iceberg `long` and Athena will read it fine.
2. If it is meant to be a timestamp, write the data so the parquet carries
int64 micros with the timestamp logical annotation (Hudi 1.x has improved
timestamp handling — tracked in #672).
Recommend linking this to #672, as both stem from Hudi writing a bare
`INT64` for a logically-timestamp field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]