Re: Relative paths and location resolution

[samuel pacheco cantu via dev]

So yes,  we do use unique IDs. The only two problems I see with this
approach are that we would need to maintain a mapping of IDs to
availability, which would be much more memory-intensive than a binary
bitmap (with an offset).

I was thinking if we could do any work, such as putting the sequence-id in
the path, or sequence + unique_id`. However, this would break during
retries because rewriting the sequence ID in the manifest would break the
path.

Meaning, we would require persisting a map of unique_id (hex-uuid) to
boolean instead of an array of bytes + 1 offset.

The Compute Engine would also have to load the mapping over the network and
persist it in Spark driver's memory.

So yes,  this is a viable option we are considering. We are also
considering building the mapping when loading the table by walking down the
metadata tree. This approach has other implications; specifically, you need
to go through all the manifest data-files to account for rewritten
manifests referencing older commits.So, workarounds exist, but they add
a level of complexity.

To be clear, the motivation for my forum post is to gain insights from
experts and share our use case with the community.

I believe path resolution work would be very beneficial if it could be
extensible and dynamic based on metadata from both the catalog and rich
types, rather than being an internal private method inside iceberg-core.
(Not saying that would be the case, I'm not familiar with the approach.)

thanks,

Sam.







On Tue, Jun 2, 2026 at 1:33 PM Ryan Blue  wrote:

> Sam, if you're already embedding IDs in your paths, why not use that
> directly? My guess is that you want unique IDs for grouping paths, but more
> sequence numbers for the actual replication work. But since you're already
> accounting for
> 
> Sam, if you're already embedding IDs in your paths, why not use that
> directly?
>
> My guess is that you want unique IDs for grouping paths, but more sequence
> numbers for the actual replication work. But since you're already
> accounting for replication that doesn't complete in order, I think you
> could change your tracking to use the IDs directly.
>
> On Tue, Jun 2, 2026 at 12:01 PM samuel pacheco cantu via dev <
> dev@iceberg.apache.org> wrote:
>
>> Yes,  it is a different problem,   we're exploring backing the
>> 'replicated' sequence ID in the backing store with the metadata pointer.
>>
>> I do understand relative paths are a simpler problem to solve, but it's
>> not clear how they unblock replication, specially real time.
>>
>> Replication requires path resolution. Given that replication is
>> asynchronous, can span multiple regions, and can be added or moved, how can
>> we signal to Iceberg that a 'sequence-id' is available in a specific
>> region?
>>
>> You are exactly right, Yufei: we are looking for a native solution to
>> keep track of which sequence-id exists in which regions.
>>
>> We currently keep it on the backing store with the metadata_pointer. As
>> replication occurs, we update the backing store to reflect bitmap
>> availability. The catalog retrieves both the metadata_pointer and the
>> bitmap,  this is how we've been thinking about solving this problem on our
>> end.  However, resolving the actual path by looking up the bitmap from the
>> catalog is not straightforward.
>>
>> The metadata (and data) files can contain relative paths, allowing copies
>> across regions without doing any rewrites, and the replication complexity
>> falls back to keeping track of "what is replicated, and where" and path
>> resolution.
>>
>> On Tue, Jun 2, 2026 at 9:26 AM Yufei Gu  wrote:
>>
>>> This sounds like a slightly different problem to me. Sharing more
>>> context, the rewrite_table_path procedure[1] already has machinery to
>>> reason about the delta between snapshots and prepare a file copy plan for
>>> table replication. That seems closer
>>> This sounds like a slightly different problem to me. Sharing more
>>> context, the rewrite_table_path procedure[1] already has machinery to
>>> reason about the delta between snapshots and prepare a file copy plan for
>>> table replication. That seems closer to the granularity Samuel is asking
>>> for than path resolution itself.
>>>
>>> I wonder if the better solution is to have an additional replication
>>> tool that tracks which snapshots or sequence ranges have been copied to
>>> which locations, similar in spirit to rewrite_table_path. That tool could
>>> own the replication state and produce copy plans, or region-specific table
>>> locations. Then relative path resolution can remain simple, while more
>>> advanced replication logic stays outside the core FileIO path resolution
>>> layer.
>>>
>>> 1.
>>> https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_table_path
>>> 

Re: Relative paths and location resolution

[samuel pacheco cantu via dev]

Yes,  it is a different problem,   we're exploring backing the 'replicated'
sequence ID in the backing store with the metadata pointer.

I do understand relative paths are a simpler problem to solve, but it's not
clear how they unblock replication, specially real time.

Replication requires path resolution. Given that replication is
asynchronous, can span multiple regions, and can be added or moved, how can
we signal to Iceberg that a 'sequence-id' is available in a specific
region?

You are exactly right, Yufei: we are looking for a native solution to keep
track of which sequence-id exists in which regions.

We currently keep it on the backing store with the metadata_pointer. As
replication occurs, we update the backing store to reflect bitmap
availability. The catalog retrieves both the metadata_pointer and the
bitmap,  this is how we've been thinking about solving this problem on our
end.  However, resolving the actual path by looking up the bitmap from the
catalog is not straightforward.

The metadata (and data) files can contain relative paths, allowing copies
across regions without doing any rewrites, and the replication complexity
falls back to keeping track of "what is replicated, and where" and path
resolution.

On Tue, Jun 2, 2026 at 9:26 AM Yufei Gu  wrote:

> This sounds like a slightly different problem to me. Sharing more context,
> the rewrite_table_path procedure[1] already has machinery to reason about
> the delta between snapshots and prepare a file copy plan for table
> replication. That seems closer
> 
> This sounds like a slightly different problem to me. Sharing more context,
> the rewrite_table_path procedure[1] already has machinery to reason about
> the delta between snapshots and prepare a file copy plan for table
> replication. That seems closer to the granularity Samuel is asking for than
> path resolution itself.
>
> I wonder if the better solution is to have an additional replication tool
> that tracks which snapshots or sequence ranges have been copied to which
> locations, similar in spirit to rewrite_table_path. That tool could own the
> replication state and produce copy plans, or region-specific table
> locations. Then relative path resolution can remain simple, while more
> advanced replication logic stays outside the core FileIO path resolution
> layer.
>
> 1.
> https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_table_path
> 
>
> Yufei
>
>
> On Tue, Jun 2, 2026 at 8:51 AM samuel pacheco cantu via dev <
> dev@iceberg.apache.org> wrote:
>
>> To give more details, yes,  the idea is that at the moment of staging a
>> commit, we can create a directory under warehouse_location/commit_id/ that
>> contains all data and metadata for that commit. Once committed,  we can
>> start replicating that directory.  Each commit_id would map to a
>> sequence-id.
>>
>> With this design, it's possible to partially replicate commits across
>> multiple regions by using a bitmap to determine if sequence-id N has
>> already been replicated to region X.
>>
>> Regarding the writer path construct,  yeah I see the LocationProvider is
>> in charge of creating the path. My question is about the FileIO input
>> files: does it make sense to keep the metadata as relative paths (or
>> absolute paths), and simply swap the files we read when getting the input
>> files? This might be a hacky and inelegant solution.  I'm reaching out to
>> understand which layer will be in charge of resolving the paths.
>>
>> The current relative path solution sounds like it would add checks within
>> the parsing code to see if the URI is relative, and then join the
>> table_location with the relative_path if applicable. I'm curious if, at the
>> very least, it would make sense to make it extendable by considering more
>> metadata.  For our use case, we are exploring the sequence ID as part of
>> the routing.
>>
>> On Mon, Jun 1, 2026 at 5:22 PM Steven Wu  wrote:
>>
>>> It seems that Sam wants a table to hold data files from multiple live
>>> regions (prefixes). The current design only supports a single prefix. On
>>> Mon, Jun 1, 2026 at 3: 19 PM Daniel Weeks  wrote:
>>> Hey Sam, I'm not sure
>>> It seems that Sam wants a table to hold data files from multiple live
>>> regions (prefixes). The current design only supports a single prefix.
>>>
>>> On Mon, Jun 1, 2026 at 3:19 PM Daniel Weeks  wrote:
>>>
 Hey Sam,

 I'm not sure I fully understand the scenario you're describing, but
 relative paths the basic concept is that you have a table location
 (provided by a catalog) and files are resolved relative to that table
 location.

 Some example are provided in the spec
 

Re: Relative paths and location resolution

[Yufei Gu]

This sounds like a slightly different problem to me. Sharing more context,
the rewrite_table_path procedure[1] already has machinery to reason about
the delta between snapshots and prepare a file copy plan for table
replication. That seems closer to the granularity Samuel is asking for than
path resolution itself.

I wonder if the better solution is to have an additional replication tool
that tracks which snapshots or sequence ranges have been copied to which
locations, similar in spirit to rewrite_table_path. That tool could own the
replication state and produce copy plans, or region-specific table
locations. Then relative path resolution can remain simple, while more
advanced replication logic stays outside the core FileIO path resolution
layer.

1.
https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_table_path

Yufei


On Tue, Jun 2, 2026 at 8:51 AM samuel pacheco cantu via dev <
dev@iceberg.apache.org> wrote:

> To give more details, yes,  the idea is that at the moment of staging a
> commit, we can create a directory under warehouse_location/commit_id/ that
> contains all data and metadata for that commit. Once committed,  we can
> start replicating that directory.  Each commit_id would map to a
> sequence-id.
>
> With this design, it's possible to partially replicate commits across
> multiple regions by using a bitmap to determine if sequence-id N has
> already been replicated to region X.
>
> Regarding the writer path construct,  yeah I see the LocationProvider is
> in charge of creating the path. My question is about the FileIO input
> files: does it make sense to keep the metadata as relative paths (or
> absolute paths), and simply swap the files we read when getting the input
> files? This might be a hacky and inelegant solution.  I'm reaching out to
> understand which layer will be in charge of resolving the paths.
>
> The current relative path solution sounds like it would add checks within
> the parsing code to see if the URI is relative, and then join the
> table_location with the relative_path if applicable. I'm curious if, at the
> very least, it would make sense to make it extendable by considering more
> metadata.  For our use case, we are exploring the sequence ID as part of
> the routing.
>
> On Mon, Jun 1, 2026 at 5:22 PM Steven Wu  wrote:
>
>> It seems that Sam wants a table to hold data files from multiple live
>> regions (prefixes). The current design only supports a single prefix. On
>> Mon, Jun 1, 2026 at 3: 19 PM Daniel Weeks  wrote:
>> Hey Sam, I'm not sure
>> It seems that Sam wants a table to hold data files from multiple live
>> regions (prefixes). The current design only supports a single prefix.
>>
>> On Mon, Jun 1, 2026 at 3:19 PM Daniel Weeks  wrote:
>>
>>> Hey Sam,
>>>
>>> I'm not sure I fully understand the scenario you're describing, but
>>> relative paths the basic concept is that you have a table location
>>> (provided by a catalog) and files are resolved relative to that table
>>> location.
>>>
>>> Some example are provided in the spec
>>> 
>>> .
>>>
>>>
>>>1.
>>>
>>>What is the intended use case for relative paths in Iceberg? Is it
>>>designed primarily for DR/replication scenarios?  What about real-time
>>>replication?
>>>
>>> The design accommodates DR/replication with proper catalog
>>> implementations to route or provide the table location.  The act of
>>> replicating the files is left out of the spec, but can be realtime
>>> depending on the implementation.
>>>
>>>1. At what point can a manifest or data file's relative path be
>>>resolved to an absolute path? Does the current design assume all 
>>> referenced
>>>data is already available locally?
>>>
>>> Paths are resolved when they're read out of manifests.  If you have a
>>> reference in metadata to a file, it should exist or readers will fail when
>>> fetching the file.  By the time you perform a commit operation, it must be
>>> referenceable.
>>>
>>>1. In FileIO, newInputFile(String path) takes a raw path string. Is
>>>there a planned mechanism to provide additional metadata (like sequence
>>>context) to help resolve paths in more complex topologies?
>>>
>>> A writer can construct paths in any way they want. Reference
>>> implementation behaviors are described in the appendix section, but there's
>>> no requirement for how they're constructed.  Relative path support is still
>>> being added to the reference implementation, but path construction is
>>> largely the responsibility of LocationProvider.  The path logic focuses on
>>> resolving or relativizing paths, not constructing them.
>>>
>>> -Dan
>>>
>>>
>>> On Mon, Jun 1, 2026 at 12:28 PM samuel pacheco cantu via dev <
>>> dev@iceberg.apache.org> wrote:
>>>
 Helo everyone,

 I have a question about relative-path resolu

Re: Relative paths and location resolution

[samuel pacheco cantu via dev]

To give more details, yes,  the idea is that at the moment of staging a
commit, we can create a directory under warehouse_location/commit_id/ that
contains all data and metadata for that commit. Once committed,  we can
start replicating that directory.  Each commit_id would map to a
sequence-id.

With this design, it's possible to partially replicate commits across
multiple regions by using a bitmap to determine if sequence-id N has
already been replicated to region X.

Regarding the writer path construct,  yeah I see the LocationProvider is in
charge of creating the path. My question is about the FileIO input files:
does it make sense to keep the metadata as relative paths (or absolute
paths), and simply swap the files we read when getting the input
files? This might be a hacky and inelegant solution.  I'm reaching out to
understand which layer will be in charge of resolving the paths.

The current relative path solution sounds like it would add checks within
the parsing code to see if the URI is relative, and then join the
table_location with the relative_path if applicable. I'm curious if, at the
very least, it would make sense to make it extendable by considering more
metadata.  For our use case, we are exploring the sequence ID as part of
the routing.

On Mon, Jun 1, 2026 at 5:22 PM Steven Wu  wrote:

> It seems that Sam wants a table to hold data files from multiple live
> regions (prefixes). The current design only supports a single prefix. On
> Mon, Jun 1, 2026 at 3: 19 PM Daniel Weeks  wrote:
> Hey Sam, I'm not sure
> 
> It seems that Sam wants a table to hold data files from multiple live
> regions (prefixes). The current design only supports a single prefix.
>
> On Mon, Jun 1, 2026 at 3:19 PM Daniel Weeks  wrote:
>
>> Hey Sam,
>>
>> I'm not sure I fully understand the scenario you're describing, but
>> relative paths the basic concept is that you have a table location
>> (provided by a catalog) and files are resolved relative to that table
>> location.
>>
>> Some example are provided in the spec
>> 
>> .
>>
>>
>>1.
>>
>>What is the intended use case for relative paths in Iceberg? Is it
>>designed primarily for DR/replication scenarios?  What about real-time
>>replication?
>>
>> The design accommodates DR/replication with proper catalog
>> implementations to route or provide the table location.  The act of
>> replicating the files is left out of the spec, but can be realtime
>> depending on the implementation.
>>
>>1. At what point can a manifest or data file's relative path be
>>resolved to an absolute path? Does the current design assume all 
>> referenced
>>data is already available locally?
>>
>> Paths are resolved when they're read out of manifests.  If you have a
>> reference in metadata to a file, it should exist or readers will fail when
>> fetching the file.  By the time you perform a commit operation, it must be
>> referenceable.
>>
>>1. In FileIO, newInputFile(String path) takes a raw path string. Is
>>there a planned mechanism to provide additional metadata (like sequence
>>context) to help resolve paths in more complex topologies?
>>
>> A writer can construct paths in any way they want. Reference
>> implementation behaviors are described in the appendix section, but there's
>> no requirement for how they're constructed.  Relative path support is still
>> being added to the reference implementation, but path construction is
>> largely the responsibility of LocationProvider.  The path logic focuses on
>> resolving or relativizing paths, not constructing them.
>>
>> -Dan
>>
>>
>> On Mon, Jun 1, 2026 at 12:28 PM samuel pacheco cantu via dev <
>> dev@iceberg.apache.org> wrote:
>>
>>> Helo everyone,
>>>
>>> I have a question about relative-path resolution in the context of
>>> multi-region replication.
>>>
>>> *Context:* We have a use case where data files may reside in different
>>> storage locations depending on the replication state. To resolve a relative
>>> path, we'd need additional context (e.g., the commit's sequence-id) to
>>> determine which region/scheme a given file should resolve to.
>>>
>>> We are actually thinking about swapping the absolute path scheme while
>>> we wait for relative-path support.  We plan to do this at the FileIO layer
>>> when requesting new input files.
>>>
>>> The problem we've got on doing the swap at the FileIO is that there are
>>> raw string path calls without not context to do any routing decision.  I
>>> would expect the same problem to occur here for relative-paths where there
>>> isn't enough context to determine the scheme.  The same argument can be
>>> made that we require even more metadata to support more complicated
>>> use-cases, such as sequence-id (and/or data-sequence-id) .
>>>
>>>
>>> *Questions:*
>>>
>>>1.
>>>
>>>Wh

Re: Relative paths and location resolution

[Steven Wu]

It seems that Sam wants a table to hold data files from multiple live
regions (prefixes). The current design only supports a single prefix.

On Mon, Jun 1, 2026 at 3:19 PM Daniel Weeks  wrote:

> Hey Sam,
>
> I'm not sure I fully understand the scenario you're describing, but
> relative paths the basic concept is that you have a table location
> (provided by a catalog) and files are resolved relative to that table
> location.
>
> Some example are provided in the spec
> .
>
>
>1.
>
>What is the intended use case for relative paths in Iceberg? Is it
>designed primarily for DR/replication scenarios?  What about real-time
>replication?
>
> The design accommodates DR/replication with proper catalog implementations
> to route or provide the table location.  The act of replicating the files
> is left out of the spec, but can be realtime depending on the
> implementation.
>
>1. At what point can a manifest or data file's relative path be
>resolved to an absolute path? Does the current design assume all referenced
>data is already available locally?
>
> Paths are resolved when they're read out of manifests.  If you have a
> reference in metadata to a file, it should exist or readers will fail when
> fetching the file.  By the time you perform a commit operation, it must be
> referenceable.
>
>1. In FileIO, newInputFile(String path) takes a raw path string. Is
>there a planned mechanism to provide additional metadata (like sequence
>context) to help resolve paths in more complex topologies?
>
> A writer can construct paths in any way they want. Reference
> implementation behaviors are described in the appendix section, but there's
> no requirement for how they're constructed.  Relative path support is still
> being added to the reference implementation, but path construction is
> largely the responsibility of LocationProvider.  The path logic focuses on
> resolving or relativizing paths, not constructing them.
>
> -Dan
>
>
> On Mon, Jun 1, 2026 at 12:28 PM samuel pacheco cantu via dev <
> dev@iceberg.apache.org> wrote:
>
>> Helo everyone,
>>
>> I have a question about relative-path resolution in the context of
>> multi-region replication.
>>
>> *Context:* We have a use case where data files may reside in different
>> storage locations depending on the replication state. To resolve a relative
>> path, we'd need additional context (e.g., the commit's sequence-id) to
>> determine which region/scheme a given file should resolve to.
>>
>> We are actually thinking about swapping the absolute path scheme while we
>> wait for relative-path support.  We plan to do this at the FileIO layer
>> when requesting new input files.
>>
>> The problem we've got on doing the swap at the FileIO is that there are
>> raw string path calls without not context to do any routing decision.  I
>> would expect the same problem to occur here for relative-paths where there
>> isn't enough context to determine the scheme.  The same argument can be
>> made that we require even more metadata to support more complicated
>> use-cases, such as sequence-id (and/or data-sequence-id) .
>>
>>
>> *Questions:*
>>
>>1.
>>
>>What is the intended use case for relative paths in Iceberg? Is it
>>designed primarily for DR/replication scenarios?  What about real-time
>>replication?
>>2. At what point can a manifest or data file's relative path be
>>resolved to an absolute path? Does the current design assume all 
>> referenced
>>data is already available locally?
>>3. In FileIO, newInputFile(String path) takes a raw path string. Is
>>there a planned mechanism to provide additional metadata (like sequence
>>context) to help resolve paths in more complex topologies?
>>
>> We'd like to understand Iceberg's direction on relative-path resolution
>> so we can align our approach with the community rather than diverging.
>>
>>
>> Thanks,
>> Sam
>>
>>

Re: Relative paths and location resolution

[Daniel Weeks]

Hey Sam,

I'm not sure I fully understand the scenario you're describing, but
relative paths the basic concept is that you have a table location
(provided by a catalog) and files are resolved relative to that table
location.

Some example are provided in the spec
.


   1.

   What is the intended use case for relative paths in Iceberg? Is it
   designed primarily for DR/replication scenarios?  What about real-time
   replication?

The design accommodates DR/replication with proper catalog implementations
to route or provide the table location.  The act of replicating the files
is left out of the spec, but can be realtime depending on the
implementation.

   1. At what point can a manifest or data file's relative path be resolved
   to an absolute path? Does the current design assume all referenced data is
   already available locally?

Paths are resolved when they're read out of manifests.  If you have a
reference in metadata to a file, it should exist or readers will fail when
fetching the file.  By the time you perform a commit operation, it must be
referenceable.

   1. In FileIO, newInputFile(String path) takes a raw path string. Is
   there a planned mechanism to provide additional metadata (like sequence
   context) to help resolve paths in more complex topologies?

A writer can construct paths in any way they want. Reference implementation
behaviors are described in the appendix section, but there's no requirement
for how they're constructed.  Relative path support is still being added to
the reference implementation, but path construction is largely the
responsibility of LocationProvider.  The path logic focuses on resolving or
relativizing paths, not constructing them.

-Dan


On Mon, Jun 1, 2026 at 12:28 PM samuel pacheco cantu via dev <
dev@iceberg.apache.org> wrote:

> Helo everyone,
>
> I have a question about relative-path resolution in the context of
> multi-region replication.
>
> *Context:* We have a use case where data files may reside in different
> storage locations depending on the replication state. To resolve a relative
> path, we'd need additional context (e.g., the commit's sequence-id) to
> determine which region/scheme a given file should resolve to.
>
> We are actually thinking about swapping the absolute path scheme while we
> wait for relative-path support.  We plan to do this at the FileIO layer
> when requesting new input files.
>
> The problem we've got on doing the swap at the FileIO is that there are
> raw string path calls without not context to do any routing decision.  I
> would expect the same problem to occur here for relative-paths where there
> isn't enough context to determine the scheme.  The same argument can be
> made that we require even more metadata to support more complicated
> use-cases, such as sequence-id (and/or data-sequence-id) .
>
>
> *Questions:*
>
>1.
>
>What is the intended use case for relative paths in Iceberg? Is it
>designed primarily for DR/replication scenarios?  What about real-time
>replication?
>2. At what point can a manifest or data file's relative path be
>resolved to an absolute path? Does the current design assume all referenced
>data is already available locally?
>3. In FileIO, newInputFile(String path) takes a raw path string. Is
>there a planned mechanism to provide additional metadata (like sequence
>context) to help resolve paths in more complex topologies?
>
> We'd like to understand Iceberg's direction on relative-path resolution so
> we can align our approach with the community rather than diverging.
>
>
> Thanks,
> Sam
>
>