cclcr opened a new issue, #17794: URL: https://github.com/apache/iceberg/issues/17794
## Problem / current limitation `RewriteTablePathUtil` / `RewriteTablePathSparkAction.rewriteLocationPrefix(sourcePrefix, targetPrefix)` assume a table's metadata files and data files all live under one common source root that maps 1:1 onto one target root. In practice this doesn't always hold: a table's metadata can live under a different physical root than its data (different buckets/storage accounts, a `write.metadata.path`/`write.data.path` override that diverges from `location`, or a cross-cloud/cross-DC migration where metadata is staged separately from data). In these cases there is no single `(source, target)` prefix pair that correctly rewrites every path referenced by table metadata (manifest lists, manifests, data files, delete files, statistics files, `write.metadata.path`/`write.data.path` properties, etc). ## Use case At Datadog we run a cross-datacenter Iceberg table sync job that copies a table's on-disk state (metadata + data) between two datacenters/clouds via `RewriteTablePath`, producing rewritten metadata pointing at the destination. Some of our tables have metadata staged under one root/bucket while the data files live under a separate root/bucket. The existing single-prefix API can't express this without either forcing everything under one prefix (not always possible) or maintaining a private fork of these classes. ## Proposed feature Generalize the prefix resolution used by `RewriteTablePathUtil`/`RewriteTablePathSparkAction` to accept: - an optional metadata-specific `(metadataSourcePrefix, metadataTargetPrefix)` pair - an optional data-specific `(dataSourcePrefix, dataTargetPrefix)` pair - the existing default `(sourcePrefix, targetPrefix)` pair as a fallback When rewriting any path, the resolver picks the most specific matching prefix (longest source-prefix match among metadata/data/default), falling back to the default pair — so `rewriteLocationPrefix(source, target)` keeps working unchanged for the existing single-pair case. We already have a working prototype against Iceberg 1.10.1: - A `PathPrefixRewrite` resolver with `single(...)` (matches current behavior) and a new `dual(defaultSource, defaultTarget, metaSource, metaTarget, dataSource, dataTarget)` factory, sharing longest-prefix-match resolution across `isKnownPath`, `relativize`, `rewritePath`, `matchedPair`. - `RewriteTablePathSparkAction` extended with a `rewritePrefixes(PathPrefixRewrite)` action method alongside the existing `rewriteLocationPrefix(String, String)`, with all internal call sites (manifest list, manifest, position/equality delete rewrite, table metadata/property rewrite) switched to use the resolver. We'd like to contribute this back as a PR if the community is open to the direction — happy to adjust the API shape (naming, whether this lives in `RewriteTablePathUtil` directly vs. a separate strategy object) based on maintainer feedback. ## Query engine Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
