haizhou-zhao commented on issue #11103:
URL: https://github.com/apache/iceberg/issues/11103#issuecomment-2341544274
cc: @rdblue (since you authored the original LoadTableResponse class)
This message is centered on issue 2. Summarizing the problem, when obtaining
TableMetadata from LoadTableResponse, the TableMetadata is reconstructed,
during which, the last-update-ms information is lost (and replaced with
`System.currentMillis()` on client side). This has a impact because
last-update-ms is returned as part of the response for `spark.sql("SELECT *
from ${db}.${table}.metadata_log_entries")`.
Code ref:
https://github.com/apache/iceberg/blob/afda8be/core/src/main/java/org/apache/iceberg/rest/responses/LoadTableResponse.java#L64
https://github.com/apache/iceberg/blob/113c6e7/core/src/main/java/org/apache/iceberg/TableMetadata.java#L938
There are 2 ways forward that I could think of:
1. When obtaining TableMetadata from LoadTableResponse, don't re-build
metadata, just return the underlying metadata as-is
2. When rebuilding TableMetadata, preserve last-update-ms field.
Let me know if the issue and the 2 options proposed make sense. Or if
there's a better way forward. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
