tanmayrauth commented on issue #1097: URL: https://github.com/apache/iceberg-go/issues/1097#issuecomment-4927664806
Good question, and it ties directly back to what we worked through in #1096. First, the cross-scheme part (s3://→file://): that's not something RenameTable can express. A Hadoop catalog instance is bound to a single warehouse root and a single FileIO resolved from that root's scheme (same as we set up in #1096). The from/to args are namespace+name path segments resolved against that one warehouse — they're not full URIs. So both sides always share the catalog's scheme; there's no seam to rename from an s3 table to a file table. Moving a table between providers is two catalog instances plus an actual byte copy, RenameTable neither crosses schemes nor transfers files, so it wouldn't help "in advance of" the transfer either. Second, unlike every other catalog, rename here isn't a pointer swap. In REST/SQL/Glue/Hive, RenameTable just updates the catalog's index entry and no data moves. The Hadoop catalog has no index, a table's identity is its path under the warehouse (and CreateTable even rejects custom locations). So "rename" necessarily means physically relocating the table directory. That's precisely why Java punts. On guarding it behind allow-unsafe-commits: directionally that's the consistent choice given the precedent we set. But note what it would be gating. The load-bearing op is Rename, the same primitive we already flagged as non-atomic on blob stores (copy+delete, crash mid-op leaves a half state). A table rename is a directory-level move across many objects, which is strictly worse than the single-file rename in CommitTable. So it inherits — and amplifies — the exact correctness gap we deferred to the concurrent-write follow-up, rather than being covered by that flag. There's also a metadata trap: Iceberg metadata references manifests and data files by absolute path. Moving the directory without rewriting the metadata tree leaves those pointers dangling, so a correct rename is effectively a copy + full metadata rewrite, not a cheap move. (Interestingly this is exactly where your downstream relative-path work would help — with relative paths a same-warehouse move stops breaking internal references.) So my take: if we add RenameTable to the Hadoop catalog, scope it narrowly to same-warehouse, same-scheme moves, gate it behind allow-unsafe-commits, and treat it as blocked on the same atomicity work as the concurrent-write follow-up, not shipped ahead of it. For your actual goals (cross-provider moves, subsetting for local debugging), a dedicated copy/migrate flow across two catalogs that rewrites metadata paths is the right shape; RenameTable isn't the vehicle for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
