So yes, we do use unique IDs. The only two problems I see with this approach are that we would need to maintain a mapping of IDs to availability, which would be much more memory-intensive than a binary bitmap (with an offset).
I was thinking if we could do any work, such as putting the sequence-id in the path, or sequence + unique_id`. However, this would break during retries because rewriting the sequence ID in the manifest would break the path. Meaning, we would require persisting a map of unique_id (hex-uuid) to boolean instead of an array of bytes + 1 offset. The Compute Engine would also have to load the mapping over the network and persist it in Spark driver's memory. So yes, this is a viable option we are considering. We are also considering building the mapping when loading the table by walking down the metadata tree. This approach has other implications; specifically, you need to go through all the manifest data-files to account for rewritten manifests referencing older commits. So, workarounds exist, but they add a level of complexity. To be clear, the motivation for my forum post is to gain insights from experts and share our use case with the community. I believe path resolution work would be very beneficial if it could be extensible and dynamic based on metadata from both the catalog and rich types, rather than being an internal private method inside iceberg-core. (Not saying that would be the case, I'm not familiar with the approach.) thanks, Sam. On Tue, Jun 2, 2026 at 1:33 PM Ryan Blue <[email protected]> wrote: > Sam, if you're already embedding IDs in your paths, why not use that > directly? My guess is that you want unique IDs for grouping paths, but more > sequence numbers for the actual replication work. But since you're already > accounting for > > Sam, if you're already embedding IDs in your paths, why not use that > directly? > > My guess is that you want unique IDs for grouping paths, but more sequence > numbers for the actual replication work. But since you're already > accounting for replication that doesn't complete in order, I think you > could change your tracking to use the IDs directly. > > On Tue, Jun 2, 2026 at 12:01 PM samuel pacheco cantu via dev < > [email protected]> wrote: > >> Yes, it is a different problem, we're exploring backing the >> 'replicated' sequence ID in the backing store with the metadata pointer. >> >> I do understand relative paths are a simpler problem to solve, but it's >> not clear how they unblock replication, specially real time. >> >> Replication requires path resolution. Given that replication is >> asynchronous, can span multiple regions, and can be added or moved, how can >> we signal to Iceberg that a 'sequence-id' is available in a specific >> region? >> >> You are exactly right, Yufei: we are looking for a native solution to >> keep track of which sequence-id exists in which regions. >> >> We currently keep it on the backing store with the metadata_pointer. As >> replication occurs, we update the backing store to reflect bitmap >> availability. The catalog retrieves both the metadata_pointer and the >> bitmap, this is how we've been thinking about solving this problem on our >> end. However, resolving the actual path by looking up the bitmap from the >> catalog is not straightforward. >> >> The metadata (and data) files can contain relative paths, allowing copies >> across regions without doing any rewrites, and the replication complexity >> falls back to keeping track of "what is replicated, and where" and path >> resolution. >> >> On Tue, Jun 2, 2026 at 9:26 AM Yufei Gu <[email protected]> wrote: >> >>> This sounds like a slightly different problem to me. Sharing more >>> context, the rewrite_table_path procedure[1] already has machinery to >>> reason about the delta between snapshots and prepare a file copy plan for >>> table replication. That seems closer >>> This sounds like a slightly different problem to me. Sharing more >>> context, the rewrite_table_path procedure[1] already has machinery to >>> reason about the delta between snapshots and prepare a file copy plan for >>> table replication. That seems closer to the granularity Samuel is asking >>> for than path resolution itself. >>> >>> I wonder if the better solution is to have an additional replication >>> tool that tracks which snapshots or sequence ranges have been copied to >>> which locations, similar in spirit to rewrite_table_path. That tool could >>> own the replication state and produce copy plans, or region-specific table >>> locations. Then relative path resolution can remain simple, while more >>> advanced replication logic stays outside the core FileIO path resolution >>> layer. >>> >>> 1. >>> https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_table_path >>> <https://urldefense.com/v3/__https://iceberg.apache.org/docs/latest/spark-procedures/*rewrite_table_path__;Iw!!Bt8RZUm9aw!6yeTeEzJ_ByuG5vTKJsg-1I92_UJOxp5vRM7Ax1cm4r4WD3a_A4JJyYDPZHW7PUuGwDgNBoJv3YU6Ag$> >>> >>> Yufei >>> >>> >>> On Tue, Jun 2, 2026 at 8:51 AM samuel pacheco cantu via dev < >>> [email protected]> wrote: >>> >>>> To give more details, yes, the idea is that at the moment of staging a >>>> commit, we can create a directory under warehouse_location/commit_id/ that >>>> contains all data and metadata for that commit. Once committed, we can >>>> start replicating that directory. Each commit_id would map to a >>>> sequence-id. >>>> >>>> With this design, it's possible to partially replicate commits across >>>> multiple regions by using a bitmap to determine if sequence-id N has >>>> already been replicated to region X. >>>> >>>> Regarding the writer path construct, yeah I see the LocationProvider >>>> is in charge of creating the path. My question is about the FileIO input >>>> files: does it make sense to keep the metadata as relative paths (or >>>> absolute paths), and simply swap the files we read when getting the input >>>> files? This might be a hacky and inelegant solution. I'm reaching out to >>>> understand which layer will be in charge of resolving the paths. >>>> >>>> The current relative path solution sounds like it would add checks >>>> within the parsing code to see if the URI is relative, and then join the >>>> table_location with the relative_path if applicable. I'm curious if, at the >>>> very least, it would make sense to make it extendable by considering more >>>> metadata. For our use case, we are exploring the sequence ID as part of >>>> the routing. >>>> >>>> On Mon, Jun 1, 2026 at 5:22 PM Steven Wu <[email protected]> wrote: >>>> >>>>> It seems that Sam wants a table to hold data files from multiple live >>>>> regions (prefixes). The current design only supports a single prefix. On >>>>> Mon, Jun 1, 2026 at 3: 19 PM Daniel Weeks <dweeks@ apache. org> >>>>> wrote: Hey Sam, I'm not sure >>>>> It seems that Sam wants a table to hold data files from multiple live >>>>> regions (prefixes). The current design only supports a single prefix. >>>>> >>>>> On Mon, Jun 1, 2026 at 3:19 PM Daniel Weeks <[email protected]> wrote: >>>>> >>>>>> Hey Sam, >>>>>> >>>>>> I'm not sure I fully understand the scenario you're describing, but >>>>>> relative paths the basic concept is that you have a table location >>>>>> (provided by a catalog) and files are resolved relative to that table >>>>>> location. >>>>>> >>>>>> Some example are provided in the spec >>>>>> <https://urldefense.com/v3/__https://iceberg.apache.org/spec/*path-resolution__;Iw!!Bt8RZUm9aw!5VJhaXv3HA50KqjLyUDUV7PikhxSsRDBSK3vP3lq783pVjyin1j9qtjPCNkQBJSm1a0xe0agLSyqe7M$> >>>>>> . >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> What is the intended use case for relative paths in Iceberg? Is >>>>>> it designed primarily for DR/replication scenarios? What about >>>>>> real-time >>>>>> replication? >>>>>> >>>>>> The design accommodates DR/replication with proper catalog >>>>>> implementations to route or provide the table location. The act of >>>>>> replicating the files is left out of the spec, but can be realtime >>>>>> depending on the implementation. >>>>>> >>>>>> 1. At what point can a manifest or data file's relative path be >>>>>> resolved to an absolute path? Does the current design assume all >>>>>> referenced >>>>>> data is already available locally? >>>>>> >>>>>> Paths are resolved when they're read out of manifests. If you have a >>>>>> reference in metadata to a file, it should exist or readers will fail >>>>>> when >>>>>> fetching the file. By the time you perform a commit operation, it must >>>>>> be >>>>>> referenceable. >>>>>> >>>>>> 1. In FileIO, newInputFile(String path) takes a raw path string. >>>>>> Is there a planned mechanism to provide additional metadata (like >>>>>> sequence >>>>>> context) to help resolve paths in more complex topologies? >>>>>> >>>>>> A writer can construct paths in any way they want. Reference >>>>>> implementation behaviors are described in the appendix section, but >>>>>> there's >>>>>> no requirement for how they're constructed. Relative path support is >>>>>> still >>>>>> being added to the reference implementation, but path construction is >>>>>> largely the responsibility of LocationProvider. The path logic focuses >>>>>> on >>>>>> resolving or relativizing paths, not constructing them. >>>>>> >>>>>> -Dan >>>>>> >>>>>> >>>>>> On Mon, Jun 1, 2026 at 12:28 PM samuel pacheco cantu via dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Helo everyone, >>>>>>> >>>>>>> I have a question about relative-path resolution in the context of >>>>>>> multi-region replication. >>>>>>> >>>>>>> *Context:* We have a use case where data files may reside in >>>>>>> different storage locations depending on the replication state. To >>>>>>> resolve >>>>>>> a relative path, we'd need additional context (e.g., the commit's >>>>>>> sequence-id) to determine which region/scheme a given file should >>>>>>> resolve >>>>>>> to. >>>>>>> >>>>>>> We are actually thinking about swapping the absolute path scheme >>>>>>> while we wait for relative-path support. We plan to do this at the >>>>>>> FileIO >>>>>>> layer when requesting new input files. >>>>>>> >>>>>>> The problem we've got on doing the swap at the FileIO is that there >>>>>>> are raw string path calls without not context to do any routing >>>>>>> decision. I >>>>>>> would expect the same problem to occur here for relative-paths where >>>>>>> there >>>>>>> isn't enough context to determine the scheme. The same argument can be >>>>>>> made that we require even more metadata to support more complicated >>>>>>> use-cases, such as sequence-id (and/or data-sequence-id) . >>>>>>> >>>>>>> >>>>>>> *Questions:* >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> What is the intended use case for relative paths in Iceberg? Is >>>>>>> it designed primarily for DR/replication scenarios? What about >>>>>>> real-time >>>>>>> replication? >>>>>>> 2. At what point can a manifest or data file's relative path be >>>>>>> resolved to an absolute path? Does the current design assume all >>>>>>> referenced >>>>>>> data is already available locally? >>>>>>> 3. In FileIO, newInputFile(String path) takes a raw path string. >>>>>>> Is there a planned mechanism to provide additional metadata (like >>>>>>> sequence >>>>>>> context) to help resolve paths in more complex topologies? >>>>>>> >>>>>>> We'd like to understand Iceberg's direction on relative-path >>>>>>> resolution so we can align our approach with the community rather than >>>>>>> diverging. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Sam >>>>>>> >>>>>>>
