LuciferYang opened a new pull request, #1014:
URL: https://github.com/apache/iceberg-cpp/pull/1014
## What
With `write.metadata.compression-codec=gzip`, metadata files are named
`<version>-<uuid>.gz.metadata.json`, but `TableMetadataUtil::Write` stored
plain JSON in them, and `Codec::FromFileName` misdetected the codec:
`find_last_of(".metadata.json")` searches for any one character of that string,
so a `.gz.metadata.json` name was classified as uncompressed. The two bugs
cancelled out inside iceberg-cpp, while files exchanged with Java failed in
both directions.
Closes #1013.
## How
- `FromFileName` uses `rfind`, matching Java's
`lastIndexOf(".metadata.json")`. Names without `.metadata.json` are now
rejected, as in Java.
- `Write` compresses the JSON when the location's codec is gzip, matching
Java's `internalWrite`. This adds a `GZipCompressor` next to `GZipDecompressor`
in `util/gzip_internal.h`, plus a `kCompressError` error kind alongside
`kDecompressError`. Inputs above zlib's 32-bit length limit are rejected
instead of being silently truncated.
The two changes are in one PR because neither works alone: fixing only the
read side would make iceberg-cpp try to inflate the plain JSON it writes today.
Tables that already have `write.metadata.compression-codec=gzip` and were
written by iceberg-cpp contain plain JSON under `.gz.metadata.json` names;
after this change iceberg-cpp reads those as gzip and fails, as Java already
does. The property defaults to `none`.
## Testing
New tests in `metadata_io_test.cc`:
- `CodecFromFileGzMetadataJson`, `CodecFromFilePlainAndLegacyGz`,
`CodecFromFileInvalidName`: codec detection for the `.gz.metadata.json`, plain,
legacy `.metadata.json.gz`, and invalid names.
- `RoundTripGzipCompressedMetadata`: writes with the gzip codec, checks the
file name, checks that the bytes on disk start with the gzip magic `1f 8b`, and
reads the metadata back.
New tests in `gzip_decompress_test.cc`: `GZipCompressorRoundTrip` (100 KiB,
more than one output buffer) and `GZipCompressorEmptyInput`.
Reverting the `rfind` change fails the three codec tests and the round trip;
removing the write-side compression fails the round trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
