[
https://issues.apache.org/jira/browse/HDDS-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-14937:
--------------------------
Description:
Iceberg tables stored in Apache Ozone traditionally(table created via ofs) use
absolute paths with the "ofs://" protocol prefix in the path. These absolute
paths prevent the table from being accessed via S3, even when a bucket link
exists.
This Epic introduces a native Ozone implementation of the Iceberg's
[RewriteTablePath
|https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
action to enable seamless protocol migration with zero data file copy. Iceberg
also provides the core util methods in
[RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
that can be used by Ozone for the same purpose.
This approach is particularly useful when integrating with REST-based catalogs
such as Apache Polaris, which expect S3-compatible locations.
We will implement the Iceberg's action and use RewriteTablePathUtil to perform
a "metadata-only" migration.
* *Traverse* the table’s metadata history.
* *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://) to
a targetPrefix (e.g., s3a:// or s3://).
* *Stage* the updated metadata files in a temporary location.
* *Perform Zero Data Copy:* The actual data files remain untouched, only the
"pointers" in the metadata are updated.
For example:
Suppose an Iceberg table is created in an Ozone volume/bucket using an ofs://
warehouse path, all file references stored across the table’s metadata
hierarchy are written as absolute ofs:// paths. This includes:
* Table metadata files (table location, manifest-list location, previous
metadata file locations)
* Manifest list files (pointing to manifest files)
* Manifest files (pointing to data files)
* Position delete files (referencing affected data files)
{panel:title=sample metadata file before rewrite }
{ "format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table{*}",
"last-sequence-number": 2,
"last-updated-ms": 1774448504371,
"last-column-id": 2,
"current-schema-id": 0,
"schemas": [...],
"default-spec-id": 0,
"partition-specs": [...],
"last-partition-id": 999,
"default-sort-order-id": 0,
"sort-orders": [...],
"properties": \{...},
"current-snapshot-id": 176890185746044789,
"refs": { "main": {
"snapshot-id": 176890185746044789,
"type": "branch"} },
"snapshots": [ { "sequence-number": 1,
"snapshot-id": 1753351619419365870,
"timestamp-ms": 1774448493051,
"summary": {
"operation": "append",
"spark.app.id": "local-1774448389815",
"added-data-files": "2",
"added-records": "2",
"added-files-size": "1224",
"changed-partition-count": "1",
"total-records": "2",
"total-files-size": "1224",
"total-data-files": "2",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "3.5.6",
"app-id": "local-1774448389815",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.10.0 (commit
2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"},
"manifest-list":
"{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro{*}",
"schema-id": 0 },
{ "sequence-number": 2,
"snapshot-id": 176890185746044789,
"parent-snapshot-id": 1753351619419365870,
"timestamp-ms": 1774448504371,
"summary": {
"operation": "append",
"spark.app.id": "local-1774448389815",
"added-data-files": "2",
"added-records": "2",
"added-files-size": "1224",
"changed-partition-count": "1",
"total-records": "4",
"total-files-size": "2448",
"total-data-files": "4",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "3.5.6",
"app-id": "local-1774448389815",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.10.0 (commit
2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"},
"manifest-list":
"{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro{*}",
"schema-id": 0} ],
"statistics": [],
"partition-statistics": [],
"snapshot-log": [...],
"metadata-log": [ {
"timestamp-ms": 1774448474465,
"metadata-file":
"{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json{*}"},
{ "timestamp-ms": 1774448493051,
"metadata-file":
"{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json{*}"
} ] }
{panel}
Since these paths are absolute ofs:// paths so they wont be resolved when we
try to access the table with REST-based catalogs such as Apache Polaris as
Polaris expects s3:// or s3a://
To make the table accessible via S3-compatible systems without copying data, we
can use Ozone native implementation of Iceberg's RewriteTablePath to rewrite
these absolute ofs:// paths to s3:// or s3a://
{panel:title=sample metadata file after rewrite }
{
"format-version": 2,
"table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
"location": "s3://buck1link/my_db/test_table",
"last-sequence-number": 2,
"last-updated-ms": 1774448504371,
"last-column-id": 2,
"current-schema-id": 0,
"schemas": [...],
"default-spec-id": 0,
"partition-specs": [...],
"last-partition-id": 999,
"default-sort-order-id": 0,
"sort-orders": [...],
"properties": \{...},
"current-snapshot-id": 176890185746044789,
"refs": {
"main": {
"snapshot-id": 176890185746044789,
"type": "branch"
}
},
"snapshots": [
{
"sequence-number": 1,
"snapshot-id": 1753351619419365870,
"timestamp-ms": 1774448493051,
"summary": {
"operation": "append",
"spark.app.id": "local-1774448389815",
"added-data-files": "2",
"added-records": "2",
"added-files-size": "1224",
"changed-partition-count": "1",
"total-records": "2",
"total-files-size": "1224",
"total-data-files": "2",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "3.5.6",
"app-id": "local-1774448389815",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.10.0 (commit
2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"
},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
"schema-id": 0
},
{
"sequence-number": 2,
"snapshot-id": 176890185746044789,
"parent-snapshot-id": 1753351619419365870,
"timestamp-ms": 1774448504371,
"summary": {
"operation": "append",
"spark.app.id": "local-1774448389815",
"added-data-files": "2",
"added-records": "2",
"added-files-size": "1224",
"changed-partition-count": "1",
"total-records": "4",
"total-files-size": "2448",
"total-data-files": "4",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "3.5.6",
"app-id": "local-1774448389815",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.10.0 (commit
2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"
},
"manifest-list":
"s3://buck1link/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
"schema-id": 0
}
],
"statistics": [],
"partition-statistics": [],
"snapshot-log": [...],
"metadata-log": [
{
"timestamp-ms": 1774448474465,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
},
{
"timestamp-ms": 1774448493051,
"metadata-file":
"s3://buck1link/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
}
]
}
{panel}
was:
Iceberg tables stored in Apache Ozone traditionally(table created via ofs) use
absolute paths with the "ofs://" protocol prefix in the path. These absolute
paths prevent the table from being accessed via S3, even when a bucket link
exists.
This Epic introduces a native Ozone implementation of the Iceberg's
[RewriteTablePath
|https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
action to enable seamless protocol migration with zero data file copy. Iceberg
also provides the core util methods in
[RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
that can be used by Ozone for the same purpose.
This approach is particularly useful when integrating with REST-based catalogs
such as Apache Polaris, which expect S3-compatible locations.
We will implement the Iceberg's action and use RewriteTablePathUtil to perform
a "metadata-only" migration.
* *Traverse* the table’s metadata history.
* *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://) to
a targetPrefix (e.g., s3a:// or s3://).
* *Stage* the updated metadata files in a temporary location.
* *Perform Zero Data Copy:* The actual data files remain untouched, only the
"pointers" in the metadata are updated.
> Ozone native implementation of Iceberg RewriteTablePath
> -------------------------------------------------------
>
> Key: HDDS-14937
> URL: https://issues.apache.org/jira/browse/HDDS-14937
> Project: Apache Ozone
> Issue Type: Epic
> Reporter: Sreeja
> Assignee: Sreeja
> Priority: Major
>
> Iceberg tables stored in Apache Ozone traditionally(table created via ofs)
> use absolute paths with the "ofs://" protocol prefix in the path. These
> absolute paths prevent the table from being accessed via S3, even when a
> bucket link exists.
> This Epic introduces a native Ozone implementation of the Iceberg's
> [RewriteTablePath
> |https://github.com/apache/iceberg/blob/1.10.x/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java]
> action to enable seamless protocol migration with zero data file copy.
> Iceberg also provides the core util methods in
> [RewriteTablePathUtil|https://github.com/apache/iceberg/blob/1.10.x/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java]
> that can be used by Ozone for the same purpose.
> This approach is particularly useful when integrating with REST-based
> catalogs such as Apache Polaris, which expect S3-compatible locations.
> We will implement the Iceberg's action and use RewriteTablePathUtil to
> perform a "metadata-only" migration.
> * *Traverse* the table’s metadata history.
> * *Rewrite* all internal absolute paths from a sourcePrefix (e.g., ofs://)
> to a targetPrefix (e.g., s3a:// or s3://).
> * *Stage* the updated metadata files in a temporary location.
> * *Perform Zero Data Copy:* The actual data files remain untouched, only the
> "pointers" in the metadata are updated.
> For example:
> Suppose an Iceberg table is created in an Ozone volume/bucket using an ofs://
> warehouse path, all file references stored across the table’s metadata
> hierarchy are written as absolute ofs:// paths. This includes:
> * Table metadata files (table location, manifest-list location, previous
> metadata file locations)
> * Manifest list files (pointing to manifest files)
> * Manifest files (pointing to data files)
> * Position delete files (referencing affected data files)
> {panel:title=sample metadata file before rewrite }
> { "format-version": 2,
> "table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
> "location": "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table{*}",
> "last-sequence-number": 2,
> "last-updated-ms": 1774448504371,
> "last-column-id": 2,
> "current-schema-id": 0,
> "schemas": [...],
> "default-spec-id": 0,
> "partition-specs": [...],
> "last-partition-id": 999,
> "default-sort-order-id": 0,
> "sort-orders": [...],
> "properties": \{...},
> "current-snapshot-id": 176890185746044789,
> "refs": { "main": {
> "snapshot-id": 176890185746044789,
> "type": "branch"} },
> "snapshots": [ { "sequence-number": 1,
> "snapshot-id": 1753351619419365870,
> "timestamp-ms": 1774448493051,
> "summary": {
> "operation": "append",
> "spark.app.id": "local-1774448389815",
> "added-data-files": "2",
> "added-records": "2",
> "added-files-size": "1224",
> "changed-partition-count": "1",
> "total-records": "2",
> "total-files-size": "1224",
> "total-data-files": "2",
> "total-delete-files": "0",
> "total-position-deletes": "0",
> "total-equality-deletes": "0",
> "engine-version": "3.5.6",
> "app-id": "local-1774448389815",
> "engine-name": "spark",
> "iceberg-version": "Apache Iceberg 1.10.0 (commit
> 2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"},
> "manifest-list":
> "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro{*}",
> "schema-id": 0 },
> { "sequence-number": 2,
> "snapshot-id": 176890185746044789,
> "parent-snapshot-id": 1753351619419365870,
> "timestamp-ms": 1774448504371,
> "summary": {
> "operation": "append",
> "spark.app.id": "local-1774448389815",
> "added-data-files": "2",
> "added-records": "2",
> "added-files-size": "1224",
> "changed-partition-count": "1",
> "total-records": "4",
> "total-files-size": "2448",
> "total-data-files": "4",
> "total-delete-files": "0",
> "total-position-deletes": "0",
> "total-equality-deletes": "0",
> "engine-version": "3.5.6",
> "app-id": "local-1774448389815",
> "engine-name": "spark",
> "iceberg-version": "Apache Iceberg 1.10.0 (commit
> 2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"},
> "manifest-list":
> "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro{*}",
> "schema-id": 0} ],
> "statistics": [],
> "partition-statistics": [],
> "snapshot-log": [...],
> "metadata-log": [ {
> "timestamp-ms": 1774448474465,
> "metadata-file":
> "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json{*}"},
> { "timestamp-ms": 1774448493051,
> "metadata-file":
> "{*}ofs://ozone-om:9862/vol1/buck1/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json{*}"
> } ] }
> {panel}
> Since these paths are absolute ofs:// paths so they wont be resolved when we
> try to access the table with REST-based catalogs such as Apache Polaris as
> Polaris expects s3:// or s3a://
> To make the table accessible via S3-compatible systems without copying data,
> we can use Ozone native implementation of Iceberg's RewriteTablePath to
> rewrite these absolute ofs:// paths to s3:// or s3a://
> {panel:title=sample metadata file after rewrite }
> {
> "format-version": 2,
> "table-uuid": "9b791462-d257-45e5-92f8-435302d2c335",
> "location": "s3://buck1link/my_db/test_table",
> "last-sequence-number": 2,
> "last-updated-ms": 1774448504371,
> "last-column-id": 2,
> "current-schema-id": 0,
> "schemas": [...],
> "default-spec-id": 0,
> "partition-specs": [...],
> "last-partition-id": 999,
> "default-sort-order-id": 0,
> "sort-orders": [...],
> "properties": \{...},
> "current-snapshot-id": 176890185746044789,
> "refs": {
> "main": {
> "snapshot-id": 176890185746044789,
> "type": "branch"
> }
> },
> "snapshots": [
> {
> "sequence-number": 1,
> "snapshot-id": 1753351619419365870,
> "timestamp-ms": 1774448493051,
> "summary": {
> "operation": "append",
> "spark.app.id": "local-1774448389815",
> "added-data-files": "2",
> "added-records": "2",
> "added-files-size": "1224",
> "changed-partition-count": "1",
> "total-records": "2",
> "total-files-size": "1224",
> "total-data-files": "2",
> "total-delete-files": "0",
> "total-position-deletes": "0",
> "total-equality-deletes": "0",
> "engine-version": "3.5.6",
> "app-id": "local-1774448389815",
> "engine-name": "spark",
> "iceberg-version": "Apache Iceberg 1.10.0 (commit
> 2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"
> },
> "manifest-list":
> "s3://buck1link/my_db/test_table/metadata/snap-1753351619419365870-1-5ac51133-8cbf-4327-bbf8-0559b463e1f9.avro",
> "schema-id": 0
> },
> {
> "sequence-number": 2,
> "snapshot-id": 176890185746044789,
> "parent-snapshot-id": 1753351619419365870,
> "timestamp-ms": 1774448504371,
> "summary": {
> "operation": "append",
> "spark.app.id": "local-1774448389815",
> "added-data-files": "2",
> "added-records": "2",
> "added-files-size": "1224",
> "changed-partition-count": "1",
> "total-records": "4",
> "total-files-size": "2448",
> "total-data-files": "4",
> "total-delete-files": "0",
> "total-position-deletes": "0",
> "total-equality-deletes": "0",
> "engine-version": "3.5.6",
> "app-id": "local-1774448389815",
> "engine-name": "spark",
> "iceberg-version": "Apache Iceberg 1.10.0 (commit
> 2114bf631e49af532d66e2ce148ee49dd1dd1f1f)"
> },
> "manifest-list":
> "s3://buck1link/my_db/test_table/metadata/snap-176890185746044789-1-5061c816-61b1-43e4-84e8-0ad689c2ea86.avro",
> "schema-id": 0
> }
> ],
> "statistics": [],
> "partition-statistics": [],
> "snapshot-log": [...],
> "metadata-log": [
> {
> "timestamp-ms": 1774448474465,
> "metadata-file":
> "s3://buck1link/my_db/test_table/metadata/00000-d480d223-a92f-4255-be8c-fef1714bb423.metadata.json"
> },
> {
> "timestamp-ms": 1774448493051,
> "metadata-file":
> "s3://buck1link/my_db/test_table/metadata/00001-3d20e8d6-e151-4442-a0d7-55533f27cf09.metadata.json"
> }
> ]
> }
> {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
