syun64 commented on issue #6978: URL: https://github.com/apache/iceberg/issues/6978#issuecomment-1451962697
I have a couple of thoughts around what the "correct" fix would be for this issue... as @szehon-ho mentioned on PR #6980 it would absolutely be nice for us to be able to support looking up metadata tables using their corresponding snapshot's schema as well. For additional context, the ability to read data using the snapshot's schema was introduced in 0.13.0 Release, which allowed users to view the actual table using the schema of the table in that point in time. This wasn't a feature that was supported for actual tables or metadata tables before. Also, the metadata table's schema can also only be updated if a user operates on a specific iceberg table across multiple versions of iceberg jar - which can happen, but only happens with a clear intention to increment the infrastructure stack in a data pipeline, instead of a running pipeline consistently and naturally invoking a schema evolution on the table. Since this is a feature regression, I think it would be very important to put in a fix to at least make the metadata table readable upon schema evolution of the actual table by conditionally using the `icebergTable.schema()` if the table is an `instanceof BaseMetadataTable`. This will at least fix this bug and revert metadata table time travel to a useable state. And then, we can continue to look into the proper way to introduce the concept of snapshot based schema look up for metadata tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
