rahulsmahadev commented on code in PR #4398:
URL: https://github.com/apache/iceberg-python/pull/4398#discussion_r4600244586
##########
mkdocs/docs/configuration.md:
##########
@@ -93,6 +93,7 @@ Iceberg tables support table properties to configure table
behavior.
| `write.summary.partition-limit` | Integer
| 0 | Maximum number of changed partitions for which
per-partition summaries are included in the snapshot summary. The default of 0
disables them |
| `write.metadata.previous-versions-max` | Integer
| 100 | The max number of previous version metadata
files to keep before deleting after commit.
|
| `write.metadata.delete-after-commit.enabled` | Boolean
| False | Whether to automatically delete old *tracked*
metadata files after each table commit. It will retain a number of the most
recent metadata files, which can be set using property
`write.metadata.previous-versions-max`. |
+| `write.metadata.compression-codec` | `{none,gzip}`
| none | Compression for the table metadata files that
PyIceberg writes. `gzip` names new metadata files `*.gz.metadata.json` and
gzips them. REST catalogs are unaffected, since the server chooses the metadata
file location. |
Review Comment:
Thanks, applied in f2e632dc. The table no longer mentions REST, and I
reworded the description: the REST server writes the metadata files, so whether
they are compressed is up to the server.
##########
pyiceberg/table/locations.py:
##########
@@ -80,12 +80,19 @@ def new_table_metadata_file_location(self, new_version: int
= 0) -> str:
str: fully-qualified URI for the new table metadata file.
Raises:
- ValueError: If the version is negative.
+ ValueError: If the version is negative, or the metadata
compression codec is not supported.
"""
+ from pyiceberg.table import TableProperties
+
if new_version < 0:
raise ValueError(f"Table metadata version: `{new_version}` must be
a non-negative integer")
- file_name = f"{new_version:05d}-{uuid.uuid4()}.metadata.json"
+ codec =
self.table_properties.get(TableProperties.METADATA_COMPRESSION,
TableProperties.METADATA_COMPRESSION_DEFAULT)
+ extension = {"none": "", "gzip": ".gz"}.get(codec.lower())
Review Comment:
Done in f2e632dc. `NONE` and `METADATA_FILE_EXTENSIONS` now sit next to
`GZIP` in `serializers.py`, the error message is built from the mapping
(`expected one of none, gzip`), and the location test checks the full message.
The import is inside `new_table_metadata_file_location`, like
`TableProperties`, because a module-level import is circular: `serializers`
imports `pyiceberg.table.metadata`, which loads `pyiceberg.table` and
`locations`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]