rahulsmahadev opened a new pull request, #4398:
URL: https://github.com/apache/iceberg-python/pull/4398

   # Rationale for this change
   
   `write.metadata.compression-codec` has no effect on the metadata file name. 
With `write.metadata.compression-codec=gzip`, PyIceberg still writes 
`00000-<uuid>.metadata.json`, while Java writes `00000-<uuid>.gz.metadata.json` 
and gzips the file. The gzip reader and writer already choose gzip from the 
`.gz.metadata.json` suffix (added in 
[apache/iceberg#7984](https://github.com/apache/iceberg/pull/7984)), so only 
the file name is missing.
   
   The Java reference is 
`BaseMetastoreTableOperations.newTableMetadataFilePath`. It reads 
`write.metadata.compression-codec` (default `none`) from the metadata being 
written, and `TableMetadataParser.getFileExtension` returns `.gz.metadata.json` 
for gzip and `.metadata.json` for none. `Codec.fromName` accepts the names 
case-insensitively and throws for any other value.
   
   This PR does the same in 
`LocationProvider.new_table_metadata_file_location`, the one place PyIceberg 
builds table metadata file names. The codec is read from the table properties. 
`none` (the default) keeps `.metadata.json`, `gzip` gives `.gz.metadata.json`, 
and any other value raises `ValueError` when a metadata file is written. The 
metastore catalogs use this for every metadata file they create or commit: SQL, 
in-memory, Hive, Glue, DynamoDB and BigQuery on create, and SQL, in-memory, 
Hive and Glue on commit (DynamoDB and BigQuery do not implement commit yet). 
The codec comes from the properties of the metadata being written, so a commit 
that sets it applies to the file that commit writes. The new 
`TableProperties.METADATA_COMPRESSION` constant is documented in 
`mkdocs/docs/configuration.md`.
   
   ```python
   from pyiceberg.table import TableProperties
   
   table = catalog.create_table(
       "db.events",
       schema=schema,
       properties={TableProperties.METADATA_COMPRESSION: "gzip"},
   )
   table.metadata_location  # .../metadata/00000-<uuid>.gz.metadata.json
   ```
   
   REST is unchanged. The REST catalog does not write table metadata files; the 
server picks the metadata location, so the client does not change the file name.
   
   ## Are these changes tested?
   
   Yes. Four new tests in `tests/catalog/test_catalog_behaviors.py` run on the 
in-memory and SQL catalogs, each with the three identifier shapes:
   
   - `test_table_writes_gzip_metadata_when_compression_codec_is_set`: the file 
name ends with `.gz.metadata.json` and the content starts with the gzip magic 
bytes. A later append keeps the codec and the version parses from the gzip name 
(`00001-`), and `load_table` returns the same metadata.
   - `test_table_metadata_compression_codec_default_and_set_on_commit`: a table 
with the default still writes `.metadata.json`. A commit that sets `gzip` makes 
the next metadata file a `.gz.metadata.json`.
   - `test_table_metadata_invalid_compression_codec_raises`: `snappy` raises 
`ValueError` on create, and no table is created. On commit it raises and the 
table is unchanged.
   - `test_table_metadata_legacy_invalid_compression_codec_on_commit`: a table 
registered with `snappy` keeps its metadata location on a commit with no 
changes. A commit that changes something raises `ValueError` and leaves the 
table unchanged, and removing the property commits normally.
   
   `tests/table/test_locations.py` adds 
`test_new_table_metadata_file_location_compression_codec` (`none`, `NONE`, 
`gzip` and `GZIP`) and 
`test_new_table_metadata_file_location_invalid_compression_codec`.
   
   The added catalog tests cover in-memory, SQL, and SQL without row-count 
support. Hive, Glue, DynamoDB and BigQuery share the filename generator; this 
PR adds no codec-specific tests to their existing unit suites, which use mocked 
services.
   
   These fail on main. Run against main with only the new `TableProperties` 
constant added, the gzip and default tests fail with `AssertionError` (the file 
name does not end with `.gz.metadata.json`), and the invalid-codec and 
legacy-codec tests fail with `DID NOT RAISE ValueError`. They pass with this 
change. The unit suite has no new failures.
   
   ## Are there any user-facing changes?
   
   Yes. With `write.metadata.compression-codec=gzip`, new table metadata files 
written by the SQL, in-memory, Hive, Glue, DynamoDB and BigQuery catalogs are 
named `*.gz.metadata.json` and gzipped. The default (`none`) does not change. 
Any other value now raises `ValueError` when PyIceberg writes a metadata file, 
so a commit that sets an unsupported value fails. For a table that already 
carries an unsupported value, remove the property or set it to `none` or `gzip` 
in the next transaction; that corrective transaction can commit normally.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to