kevinjqliu opened a new pull request, #4431:
URL: https://github.com/apache/iceberg-python/pull/4431
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
Adds end-to-end tests for encrypted tables. Spark writes encrypted v3
tables, and PyIceberg reads them. Follows the read support in #4159, #4308,
#4309, and #4326.
Spark's hive catalog is configured with `UnitestKMS`, an in-memory KMS with
static master keys from `iceberg-core`'s tests jar. PyIceberg reads with an
in-memory KMS holding the same key. The tests cover AES-128 and AES-256 data
keys:
- scans, including deletion vectors, filters, and projections
- time travel
- missing or wrong master key
- `inspect` metadata tables
`iceberg-core`'s tests jar also has a mock `S3ABlockOutputStream`, so it's
downloaded after `hadoop-aws` to keep S3A writes working.
`test_inspect_encrypted_table` depends on the `inspect.entries()` fix in
#4199.
## Are these changes tested?
Yes.
- `make test-integration` with #4199 applied: 1371 passed, 71 skipped
- `uv run prek run --files dev/spark/Dockerfile
dev/spark/spark-defaults.conf tests/integration/test_encryption.py`: passed
## Are there any user-facing changes?
No.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]