rahulsmahadev opened a new pull request, #4398: URL: https://github.com/apache/iceberg-python/pull/4398
# Rationale for this change `write.metadata.compression-codec` has no effect on the metadata file name. With `write.metadata.compression-codec=gzip`, PyIceberg still writes `00000-<uuid>.metadata.json`, while Java writes `00000-<uuid>.gz.metadata.json` and gzips the file. The gzip reader and writer already choose gzip from the `.gz.metadata.json` suffix (added in [apache/iceberg#7984](https://github.com/apache/iceberg/pull/7984)), so only the file name is missing. The Java reference is `BaseMetastoreTableOperations.newTableMetadataFilePath`. It reads `write.metadata.compression-codec` (default `none`) from the metadata being written, and `TableMetadataParser.getFileExtension` returns `.gz.metadata.json` for gzip and `.metadata.json` for none. `Codec.fromName` accepts the names case-insensitively and throws for any other value. This PR does the same in `LocationProvider.new_table_metadata_file_location`, the one place PyIceberg builds table metadata file names. The codec is read from the table properties. `none` (the default) keeps `.metadata.json`, `gzip` gives `.gz.metadata.json`, and any other value raises `ValueError` when a metadata file is written. The metastore catalogs use this for every metadata file they create or commit: SQL, in-memory, Hive, Glue, DynamoDB and BigQuery on create, and SQL, in-memory, Hive and Glue on commit (DynamoDB and BigQuery do not implement commit yet). The codec comes from the properties of the metadata being written, so a commit that sets it applies to the file that commit writes. The new `TableProperties.METADATA_COMPRESSION` constant is documented in `mkdocs/docs/configuration.md`. ```python from pyiceberg.table import TableProperties table = catalog.create_table( "db.events", schema=schema, properties={TableProperties.METADATA_COMPRESSION: "gzip"}, ) table.metadata_location # .../metadata/00000-<uuid>.gz.metadata.json ``` REST is unchanged. The REST catalog does not write table metadata files; the server picks the metadata location, so the client does not change the file name. ## Are these changes tested? Yes. Four new tests in `tests/catalog/test_catalog_behaviors.py` run on the in-memory and SQL catalogs, each with the three identifier shapes: - `test_table_writes_gzip_metadata_when_compression_codec_is_set`: the file name ends with `.gz.metadata.json` and the content starts with the gzip magic bytes. A later append keeps the codec and the version parses from the gzip name (`00001-`), and `load_table` returns the same metadata. - `test_table_metadata_compression_codec_default_and_set_on_commit`: a table with the default still writes `.metadata.json`. A commit that sets `gzip` makes the next metadata file a `.gz.metadata.json`. - `test_table_metadata_invalid_compression_codec_raises`: `snappy` raises `ValueError` on create, and no table is created. On commit it raises and the table is unchanged. - `test_table_metadata_legacy_invalid_compression_codec_on_commit`: a table registered with `snappy` keeps its metadata location on a commit with no changes. A commit that changes something raises `ValueError` and leaves the table unchanged, and removing the property commits normally. `tests/table/test_locations.py` adds `test_new_table_metadata_file_location_compression_codec` (`none`, `NONE`, `gzip` and `GZIP`) and `test_new_table_metadata_file_location_invalid_compression_codec`. The added catalog tests cover in-memory, SQL, and SQL without row-count support. Hive, Glue, DynamoDB and BigQuery share the filename generator; this PR adds no codec-specific tests to their existing unit suites, which use mocked services. These fail on main. Run against main with only the new `TableProperties` constant added, the gzip and default tests fail with `AssertionError` (the file name does not end with `.gz.metadata.json`), and the invalid-codec and legacy-codec tests fail with `DID NOT RAISE ValueError`. They pass with this change. The unit suite has no new failures. ## Are there any user-facing changes? Yes. With `write.metadata.compression-codec=gzip`, new table metadata files written by the SQL, in-memory, Hive, Glue, DynamoDB and BigQuery catalogs are named `*.gz.metadata.json` and gzipped. The default (`none`) does not change. Any other value now raises `ValueError` when PyIceberg writes a metadata file, so a commit that sets an unsupported value fails. For a table that already carries an unsupported value, remove the property or set it to `none` or `gzip` in the next transaction; that corrective transaction can commit normally. Please add the `changelog` label. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
