[
https://issues.apache.org/jira/browse/HDDS-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-14937:
--------------------------
Description:
Iceberg tables stored in Apache Ozone traditionally(table created via ofs) use
absolute paths with the "ofs://" protocol prefix in the path. These absolute
paths prevent the table from being accessed via S3, even when a bucket link
exists.
This Epic introduces a native Ozone implementation of the Iceberg's
[RewriteTablePath
|https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
action to enable seamless protocol migration with zero data file copy. Iceberg
also provides the core util methods in
[RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
that can be used by Ozone for the same purpose.
This approach is particularly useful when integrating with REST-based catalogs
such as Apache Polaris, which expect S3-compatible locations.
We will implement the Iceberg's action and use RewriteTablePathUtil to perform
a "metadata-only" migration.
* *Traverse* the table’s metadata history.
* *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://) to
a targetPrefix (e.g., s3a:// or s3://).
* *Stage* the updated metadata files in a temporary location.
* *Perform Zero Data Copy:* The actual data files remain untouched, only the
"pointers" in the metadata(metadata version file, manifest list , manifest file
, position delete file) are updated.
For example:
Suppose an Iceberg table is present in an Ozone volume/bucket using an ofs://
path say "{*}ofs://om:9862/vol1/buck1/my_db/test_table"{*}, all file references
stored across the table’s metadata hierarchy are mentioned as absolute ofs://
paths. This includes:
* Table metadata files (table location, manifest-list location, previous
metadata file locations)
* Manifest list files (pointing to manifest files)
* Manifest files (pointing to data files)
* Position delete files (referencing affected data files)
sample metadata file (before rewrite):
{code:java}
{ "format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "ofs://ozone-om:9862/vol1/buck1/my_db/test_table",
.
.
.
},
"snapshots": [{...},
"manifest-list":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
"schema-id": 0 },
{...},
"manifest-list":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
"schema-id": 0 } ],
.
.
.
"metadata-log": [
{
"timestamp-ms": 1774448474465,
"metadata-file":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
},
{
"timestamp-ms": 1774448493051,
"metadata-file":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
} ]} {code}
Now if we try to access this table via a REST based catalog like Apache Polaris
then it would fail as polaris expects s3:// or s3a://
{code:java}
org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Invalid locations
'[ofs://om:9862/vol1/buck1/my_db/test_table]' for identifier
'my_db.test_table': ofs://om:9862/vol1/buck1/my_db/test_table is not in the
list of allowed locations{code}
we won't even be able to register the table with polaris catalog as it sees
ofs:// paths in the files. Or if we use any engine that tries to access the
table via s3 it would also fail as it won't be able to resolve ofs:// paths.
h3. Rewriting paths to S3
To make the table accessible via S3-compatible systems {*}without copying data
files{*}, we use Ozone's native implementation of Iceberg's
{{RewriteTablePath}} action.
Steps:
# *Create a bucket link* in the Ozone {{/s3v}} volume pointing to the bucket
where the table exists.
# Provide the *sourcePrefix*
({{{}ofs://om:9862/vol1/buck1/my_db/test_table{}}}) and *targetPrefix*
({{{}s3://buck1link/my_db/test_table{}}}).
# Optionally, provide *start/end metadata versions* or a *staging location*
for the rewritten metadata files.
# Run the rewrite action — this updates all embedded paths in metadata version
files, manifest lists, manifests, and position delete files {*}without touching
the actual data{*}.
# Copy the rewritten metadata from the staging location back to the table’s
location (not handled automatically by Ozone's implementation).
sample metadata file (after rewrite):
{code:java}
{ "format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "s3://buck1link/my_db/test_table",
.
.
.
},
"snapshots": [{...},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
"schema-id": 0 },
{...},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
"schema-id": 0 } ],
. . .
"metadata-log": [
{
"timestamp-ms": 1774448474465,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
},
{
"timestamp-ms": 1774448493051,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
} ]} {code}
After this rewrite:
* The table is {*}accessible via S3{*}.
* It can now be *registered with Polaris* without any path-related errors.
was:
Iceberg tables stored in Apache Ozone traditionally(table created via ofs) use
absolute paths with the "ofs://" protocol prefix in the path. These absolute
paths prevent the table from being accessed via S3, even when a bucket link
exists.
This Epic introduces a native Ozone implementation of the Iceberg's
[RewriteTablePath
|https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
action to enable seamless protocol migration with zero data file copy. Iceberg
also provides the core util methods in
[RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
that can be used by Ozone for the same purpose.
This approach is particularly useful when integrating with REST-based catalogs
such as Apache Polaris, which expect S3-compatible locations.
We will implement the Iceberg's action and use RewriteTablePathUtil to perform
a "metadata-only" migration.
* *Traverse* the table’s metadata history.
* *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://) to
a targetPrefix (e.g., s3a:// or s3://).
* *Stage* the updated metadata files in a temporary location.
* *Perform Zero Data Copy:* The actual data files remain untouched, only the
"pointers" in the metadata are updated.
For example:
Suppose an Iceberg table is present in an Ozone volume/bucket using an ofs://
path say "{*}ofs://om:9862/vol1/buck1/my_db/test_table"{*}, all file references
stored across the table’s metadata hierarchy are mentioned as absolute ofs://
paths. This includes:
* Table metadata files (table location, manifest-list location, previous
metadata file locations)
* Manifest list files (pointing to manifest files)
* Manifest files (pointing to data files)
* Position delete files (referencing affected data files)
sample metadata file (before rewrite):
{code:java}
{ "format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "ofs://ozone-om:9862/vol1/buck1/my_db/test_table",
.
.
.
},
"snapshots": [{...},
"manifest-list":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
"schema-id": 0 },
{...},
"manifest-list":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
"schema-id": 0 } ],
.
.
.
"metadata-log": [
{
"timestamp-ms": 1774448474465,
"metadata-file":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
},
{
"timestamp-ms": 1774448493051,
"metadata-file":
"ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
} ]} {code}
Now if we try to access this table via a REST based catalog like Apache Polaris
then it would fail as polaris expects s3:// or s3a://
{code:java}
org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Invalid locations
'[ofs://om:9862/vol1/buck1/my_db/test_table]' for identifier
'my_db.test_table': ofs://om:9862/vol1/buck1/my_db/test_table is not in the
list of allowed locations{code}
we won't even be able to register the table with polaris catalog as it sees
ofs:// paths in the files. Or if we use any engine that tries to access the
table via s3 it would also fail as it won't be able to resolve ofs:// paths.
h3. Rewriting paths to S3
To make the table accessible via S3-compatible systems {*}without copying data
files{*}, we use Ozone's native implementation of Iceberg's
{{RewriteTablePath}} action.
Steps:
# *Create a bucket link* in the Ozone {{/s3v}} volume pointing to the bucket
where the table exists.
# Provide the *sourcePrefix*
({{{}ofs://om:9862/vol1/buck1/my_db/test_table{}}}) and *targetPrefix*
({{{}s3://buck1link/my_db/test_table{}}}).
# Optionally, provide *start/end metadata versions* or a *staging location*
for the rewritten metadata files.
# Run the rewrite action — this updates all embedded paths in metadata files,
manifest lists, manifests, and position delete files {*}without touching the
actual data{*}.
# Copy the rewritten metadata from the staging location back to the table’s
location (not handled automatically by Ozone's implementation).
sample metadata file (after rewrite):
{code:java}
{ "format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "s3://buck1link/my_db/test_table",
.
.
.
},
"snapshots": [{...},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
"schema-id": 0 },
{...},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
"schema-id": 0 } ],
. . .
"metadata-log": [
{
"timestamp-ms": 1774448474465,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
},
{
"timestamp-ms": 1774448493051,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
} ]} {code}
After this rewrite:
* The table is {*}accessible via S3{*}.
* It can now be *registered with Polaris* without any path-related errors.
> Ozone native implementation of Iceberg RewriteTablePath
> -------------------------------------------------------
>
> Key: HDDS-14937
> URL: https://issues.apache.org/jira/browse/HDDS-14937
> Project: Apache Ozone
> Issue Type: Epic
> Reporter: Sreeja
> Assignee: Sreeja
> Priority: Major
>
> Iceberg tables stored in Apache Ozone traditionally(table created via ofs)
> use absolute paths with the "ofs://" protocol prefix in the path. These
> absolute paths prevent the table from being accessed via S3, even when a
> bucket link exists.
> This Epic introduces a native Ozone implementation of the Iceberg's
> [RewriteTablePath
> |https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
> action to enable seamless protocol migration with zero data file copy.
> Iceberg also provides the core util methods in
> [RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
> that can be used by Ozone for the same purpose.
> This approach is particularly useful when integrating with REST-based
> catalogs such as Apache Polaris, which expect S3-compatible locations.
> We will implement the Iceberg's action and use RewriteTablePathUtil to
> perform a "metadata-only" migration.
> * *Traverse* the table’s metadata history.
> * *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://)
> to a targetPrefix (e.g., s3a:// or s3://).
> * *Stage* the updated metadata files in a temporary location.
> * *Perform Zero Data Copy:* The actual data files remain untouched, only the
> "pointers" in the metadata(metadata version file, manifest list , manifest
> file , position delete file) are updated.
> For example:
> Suppose an Iceberg table is present in an Ozone volume/bucket using an ofs://
> path say "{*}ofs://om:9862/vol1/buck1/my_db/test_table"{*}, all file
> references stored across the table’s metadata hierarchy are mentioned as
> absolute ofs:// paths. This includes:
> * Table metadata files (table location, manifest-list location, previous
> metadata file locations)
> * Manifest list files (pointing to manifest files)
> * Manifest files (pointing to data files)
> * Position delete files (referencing affected data files)
> sample metadata file (before rewrite):
> {code:java}
> { "format-version": 2,
> "table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
> "location": "ofs://ozone-om:9862/vol1/buck1/my_db/test_table",
> .
> .
> .
> },
> "snapshots": [{...},
> "manifest-list":
> "ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
> "schema-id": 0 },
> {...},
> "manifest-list":
> "ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
> "schema-id": 0 } ],
> .
> .
> .
> "metadata-log": [
> {
> "timestamp-ms": 1774448474465,
> "metadata-file":
> "ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
>
> },
> {
> "timestamp-ms": 1774448493051,
> "metadata-file":
> "ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
> } ]} {code}
> Now if we try to access this table via a REST based catalog like Apache
> Polaris then it would fail as polaris expects s3:// or s3a://
> {code:java}
> org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Invalid
> locations '[ofs://om:9862/vol1/buck1/my_db/test_table]' for identifier
> 'my_db.test_table': ofs://om:9862/vol1/buck1/my_db/test_table is not in the
> list of allowed locations{code}
> we won't even be able to register the table with polaris catalog as it sees
> ofs:// paths in the files. Or if we use any engine that tries to access the
> table via s3 it would also fail as it won't be able to resolve ofs:// paths.
> h3. Rewriting paths to S3
> To make the table accessible via S3-compatible systems {*}without copying
> data files{*}, we use Ozone's native implementation of Iceberg's
> {{RewriteTablePath}} action.
> Steps:
> # *Create a bucket link* in the Ozone {{/s3v}} volume pointing to the bucket
> where the table exists.
> # Provide the *sourcePrefix*
> ({{{}ofs://om:9862/vol1/buck1/my_db/test_table{}}}) and *targetPrefix*
> ({{{}s3://buck1link/my_db/test_table{}}}).
> # Optionally, provide *start/end metadata versions* or a *staging location*
> for the rewritten metadata files.
> # Run the rewrite action — this updates all embedded paths in metadata
> version files, manifest lists, manifests, and position delete files
> {*}without touching the actual data{*}.
> # Copy the rewritten metadata from the staging location back to the table’s
> location (not handled automatically by Ozone's implementation).
> sample metadata file (after rewrite):
> {code:java}
> { "format-version": 2,
> "table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
> "location": "s3://buck1link/my_db/test_table",
> .
> .
> .
> },
> "snapshots": [{...},
> "manifest-list":
> "s3://buck1link/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
> "schema-id": 0 },
> {...},
> "manifest-list":
> "s3://buck1link/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
> "schema-id": 0 } ],
> . . .
> "metadata-log": [
> {
> "timestamp-ms": 1774448474465,
> "metadata-file":
> "s3://buck1link/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
>
> },
> {
> "timestamp-ms": 1774448493051,
> "metadata-file":
> "s3://buck1link/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
> } ]} {code}
> After this rewrite:
> * The table is {*}accessible via S3{*}.
> * It can now be *registered with Polaris* without any path-related errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
