rahulsmahadev opened a new pull request, #4456:
URL: https://github.com/apache/iceberg-python/pull/4456
# Rationale for this change
Bug: `new_table_metadata` reads the requested format version with
`properties.pop("format-version", ...)` on the dict passed to `create_table`,
so the caller's dict loses the key. For example, reusing a v1 properties dict
with the SQL catalog creates the second table as v2:
```python
properties = {"format-version": "1", "owner": "etl"}
catalog.create_table("db.events_2024", schema,
properties=properties).format_version # 1
catalog.create_table("db.events_2025", schema,
properties=properties).format_version # 2
properties # {'owner': 'etl'}
```
`create_table_transaction` removes the key before the commit, and
`create_table_if_not_exists` also removes it when the table exists. The pop is
from #411, to keep `format-version` out of the persisted table properties.
Fix: pop from a copy. The Glue and DynamoDB `create_table` wrote the
caller's dict to the Glue parameters and the DynamoDB item after this call, so
they relied on the pop; they now write the new metadata's properties, as Glue's
`commit_table` does. That also turns the values into strings there: Glue stores
`True` as `true`, and DynamoDB accepts non-string values instead of failing
with `ParamValidationError`.
Java: the `BaseMetastoreCatalog` table builder copies the properties into
its own map, and `TableMetadata.newTableMetadata` reads the version with
`PropertyUtil.propertyAsInt` and persists a new filtered map, so the caller's
map is never changed.
- Java's `persistedProperties` leaves out all reserved properties (`uuid`,
`snapshot-count`, ...); `new_table_metadata` still only leaves out
`format-version`.
## Are these changes tested?
Yes, `test_create_v1_tables_with_shared_properties` in
`tests/catalog/test_catalog_behaviors.py`,
`test_new_table_metadata_does_not_change_properties` in
`tests/table/test_metadata.py` and `test_create_v1_table` in
`tests/catalog/test_dynamodb.py` fail on main (the dict loses `format-version`,
or the versions are `[1, 2, 2]`) and pass with this change.
`test_create_v1_table` and `test_create_table_s3tables` in
`tests/catalog/test_glue.py` now check the exact Glue parameters.
## Are there any user-facing changes?
Yes, `create_table` no longer changes the passed `properties` dict, and the
Glue and DynamoDB catalogs write the property values as strings when creating a
table.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]