jackye1995 commented on issue #1617: URL: https://github.com/apache/iceberg/issues/1617#issuecomment-764936126
Thanks for the design doc! I wonder if this can be done through changes in Catalog and FileIO without touching the Iceberg specification, which can be much simpler to ship without conflicting with already quite a few changes for the format v2 specifications. The key difficulty I see is that table property is a part of the table metadata file. Therefore we cannot know if a table should use any special file path override without actually reading that file. If we have a way to know it without using FileIO, then we can decide for a specific table, all files should be read through a region-specific client for replication, or a different URI scheme for backup and HDFS federation. So here is my alternative proposal: 1. add interface `TablePropertyIO` that is dedicated to read and write table properties. 2. update catalog implementations (mostly in `BaseMetastoreCatalog`) to use `TablePropertyIO` when necessary, by adding it as a part of the `TableOperations` interface similar to `FileIO io()`. 3. add new table property `table-io-impl`, that can be customized to do things like replicaiton, backup, etc. 4. implement those specific table `FileIO` classes. Could this proposal satisfy your requirements? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
