tomtongue opened a new pull request #3468:
URL: https://github.com/apache/iceberg/pull/3468
## Changes
Adding the StroageDescriptor parameter to a renamed table in the Glue Data
Catalog by ALTER TABLE RENAME TO with Iceberg.
## Current situation
The ALTER TABLE RENAME TO supported in Iceberg operates CREATE A NEW TABLE
and DROP TABLE because the Glue Data Catalog doesn’t support renaming tables.
Currently, the StorageDescriptor part is not copied when running ALTER TABLE
RENAME TO with Iceberg as the following example.
### Example
**Before the table is renamed**:
```json
➜ ~ aws glue get-table --database-name db --name iceberg_1636000905
{
"Table": {
"Name": "iceberg_1636000905",
"DatabaseName": "db",
"CreateTime": 1636000913.0,
"UpdateTime": 1636000913.0,
"Retention": 0,
"StorageDescriptor": {
"Columns": [
{
"Name": "id",
"Type": "bigint",
"Parameters": {
"iceberg.field.id": "1",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "bigint",
"iceberg.field.type.typeid": "LONG",
"iceberg.field.usage": "schema-column"
}
},
{
"Name": "data",
"Type": "string",
"Parameters": {
"iceberg.field.id": "2",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "string",
"iceberg.field.type.typeid": "STRING",
"iceberg.field.usage": "schema-column"
}
}
],
"Location": "s3://.../iceberg_1636000905",
// omitted ...
"StoredAsSubDirectories": false
},
"TableType": "EXTERNAL_TABLE",
"Parameters": {
"metadata_location":
"s3://.../iceberg_1636000905/metadata/00000-18f06867-fa8b-44b3-b25d-424b88a3b683.metadata.json",
"table_type": "ICEBERG"
},
"CreatedBy": "arn:aws:sts::account_id:role_name",
"IsRegisteredWithLakeFormation": false,
"CatalogId": "account_id",
"IsRowFilteringEnabled": false
}
}
```
**After the table is renamed**: => StorageDescriptor was missed.
```json
➜ ~ aws glue get-table --database-name db --name iceberg_1636000905_rename
{
"Table": {
"Name": "iceberg_1636000905_rename",
"DatabaseName": "db",
"CreateTime": 1636001356.0,
"UpdateTime": 1636001356.0,
"Retention": 0,
"TableType": "EXTERNAL_TABLE",
"Parameters": {
"metadata_location":
"s3://.../iceberg_1636000905/metadata/00000-18f06867-fa8b-44b3-b25d-424b88a3b683.metadata.json",
"table_type": "ICEBERG"
},
"CreatedBy": "arn:aws:sts::account_id:role_name",
"IsRegisteredWithLakeFormation": false,
"CatalogId": "account_id",
"IsRowFilteringEnabled": false
}
}
```
## After this change
The StorageDescriptor part is filled in by the iceberg renameTable operation.
### Example
**Before the table is renamed**:
```json
➜ ~ aws glue get-table --database-name db --name iceberg_1636005278
{
"Table": {
"Name": "iceberg_1636005278",
"DatabaseName": "db",
"CreateTime": 1636005285.0,
"UpdateTime": 1636005285.0,
"Retention": 0,
"StorageDescriptor": {
"Columns": [
{
"Name": "id",
"Type": "bigint",
"Parameters": {
"iceberg.field.id": "1",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "bigint",
"iceberg.field.type.typeid": "LONG",
"iceberg.field.usage": "schema-column"
}
},
{
"Name": "data",
"Type": "string",
"Parameters": {
"iceberg.field.id": "2",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "string",
"iceberg.field.type.typeid": "STRING",
"iceberg.field.usage": "schema-column"
}
}
],
"Location": "s3://.../iceberg_1636005278",
"Compressed": false,
"NumberOfBuckets": 0,
"SortColumns": [],
"StoredAsSubDirectories": false
},
"TableType": "EXTERNAL_TABLE",
"Parameters": {
"metadata_location":
"s3://.../iceberg_1636005278/metadata/00000-1b5f7c9b-01ec-42a3-88da-f62227e74e74.metadata.json",
"table_type": "ICEBERG"
},
"CreatedBy": "arn:aws:sts::account_id:role_name",
"IsRegisteredWithLakeFormation": false,
"CatalogId": "account_id",
"IsRowFilteringEnabled": false
}
}
```
**After the table is renamed**: => StorageDescriptor was copied from the
previous version of table.
```json
➜ ~ aws glue get-table --database-name db --name iceberg_1636005278_rename
{
"Table": {
"Name": "iceberg_1636005278_rename",
"DatabaseName": "db",
"CreateTime": 1636005530.0,
"UpdateTime": 1636005530.0,
"Retention": 0,
"StorageDescriptor": {
"Columns": [
{
"Name": "id",
"Type": "bigint",
"Parameters": {
"iceberg.field.id": "1",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "bigint",
"iceberg.field.type.typeid": "LONG",
"iceberg.field.usage": "schema-column"
}
},
{
"Name": "data",
"Type": "string",
"Parameters": {
"iceberg.field.id": "2",
"iceberg.field.optional": "true",
"iceberg.field.type.string": "string",
"iceberg.field.type.typeid": "STRING",
"iceberg.field.usage": "schema-column"
}
}
],
"Location": "s3://.../iceberg_1636005278",
"Compressed": false,
"NumberOfBuckets": 0,
"SortColumns": [],
"StoredAsSubDirectories": false
},
"TableType": "EXTERNAL_TABLE",
"Parameters": {
"metadata_location":
"s3://.../iceberg_1636005278/metadata/00000-1b5f7c9b-01ec-42a3-88da-f62227e74e74.metadata.json",
"table_type": "ICEBERG"
},
"CreatedBy": "arn:aws:sts::account_id:role_name",
"IsRegisteredWithLakeFormation": false,
"CatalogId": "account_id",
"IsRowFilteringEnabled": false
}
}
```
Best regards,
Tom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
