Hi everyone, Iceberg’s Load Table v2 proposal ( https://github.com/apache/iceberg/pull/17642) is moving forward. I’d like to discuss what it means for Polaris and a few opportunities we should consider.
1. Supporting Iceberg v4 The proposal adds a v2 load endpoint that supports v4 tables. The v1 endpoint would reject v4 tables to protect older clients. For Polaris, this means adding the new endpoint and supporting metadata changes such as root manifests and relative paths. 2. Storing table metadata in the metastore The proposal allows tables without a metadata JSON file location. This could let Polaris store table metadata directly in the metastore, avoiding the need to read and write a separate JSON file during loads and commits. This would be a big perf boost for every table commit and read. I’d love to see this happen. We would need a separate design for this, covering commit validation, metadata versioning, and how to export metadata when a table leaves Polaris. Then engine side changes are need as well, but it is more of Iceberg community’s concern. 3. Defaulting snapshots to refs instead of all Large table metadata has been a recurring problem for users of Polaris and other catalogs, including Glue. This has also come up in several Iceberg community discussions. I think it a good time to propose making snapshots=refs the default. Returning only snapshots referenced by branches or tags would reduce response sizes for tables with long snapshot histories. Clients could still request all when they need the full set. This would help with response size, though reducing the cost of loading metadata inside Polaris would require additional work. I suggest we start with v2 compatibility, explore metastore storage separately, and raise the snapshot default upstream while the API is still being discussed. What do people think? Are there compatibility concerns we should consider? Is there any other features in your mind for loadTable v2? Thanks, Yufei
