shangxinli opened a new issue, #988:
URL: https://github.com/apache/iceberg-cpp/issues/988
### Feature Description
**What the feature achieves:**
Adds Iceberg table encryption to iceberg-cpp, wire-compatible with the Java
implementation (`StandardEncryptionManager`). iceberg-cpp can read tables that
Java encrypted, and Java can read encrypted tables that iceberg-cpp writes.
Data files, delete files, deletion vectors, manifests and manifest lists are
encrypted; `metadata.json` is not, as in Java.
**Design: follow the Java scheme exactly**
Java defines the formats, so the C++ design matches it rather than adding
anything new:
- **Key hierarchy (envelope encryption).** The table master key stays in the
KMS (`encryption.key-id`). It wraps key encryption keys (KEKs), which are kept
in the table metadata `encryption-keys` with a `KEY_TIMESTAMP` and rotated
every 730 days. A KEK AES-GCM-encrypts the key metadata of each manifest list
(AAD = KEK timestamp); the snapshot `key-id` points to that entry. Manifest
DEKs live in the manifest list `key_metadata`, and data/delete file DEKs live
in the manifest `key_metadata`, both in plaintext inside an encrypted file.
- **Key metadata.** `StandardKeyMetadata`: version byte `1` followed by an
Avro record `{encryption_key, aad_prefix, file_length}`. The stored length is
recorded for every encrypted file, including Parquet, because Java requires it
for AGS1 reads (iceberg-rust found this in
[apache/iceberg-rust#3236](https://github.com/apache/iceberg-rust/pull/3236)).
- **File formats.** Parquet data and delete files use Parquet modular
encryption (encrypted footer, `PARE`, footer key = DEK, AAD prefix not stored
in the file). Manifests, manifest lists, Avro data files and Puffin deletion
vectors use AES GCM Streams (`AGS1`,
[spec](https://github.com/apache/iceberg/blob/main/format/gcm-stream-spec.md)).
- **Trust model.** Encryption requires format v3. The table key ID cannot
change once set. Catalogs must take the key ID from a trusted source, not only
from `metadata.json`; the REST catalog creates its KMS client from client-side
properties only.
**Library layering:** the core library `iceberg` stays free of crypto
dependencies. It defines the interfaces (`KeyManagementClient`,
`EncryptionManager`, `EncryptingFileIO`) and reaches the implementations
through registries, the same pattern as `ReaderFactoryRegistry` and
`FileIORegistry`. The bundle library `iceberg_bundle` (Arrow, Avro, Parquet)
adds OpenSSL and the standard implementation. Cloud KMS clients are an optional
library, `iceberg_kms`.
**Why this feature is needed:**
Problem: encrypted Iceberg tables written by Java (Spark, Flink, Trino with
the Java library) cannot be used from C++ engines today.
Current gaps in iceberg-cpp:
- Table metadata `encryption-keys` is parsed, but there is no snapshot
`key-id`, so encrypted snapshots cannot be resolved
- No crypto: no AES-GCM, no AGS1 streams, no key metadata, no KMS integration
- The vendored Arrow builds Parquet with encryption compiled out
(`encryption_internal_nossl.cc`)
- Writers have no guard: writing to an encrypted table would silently write
plaintext files
- Table-level encryption is listed as "not yet planned" in the v3 roadmap
([#959](https://github.com/apache/iceberg-cpp/issues/959))
Other implementations: Java has the core in main (Hive catalog wired end to
end); iceberg-rust has merged an 18-step PR series
([apache/iceberg-rust#2034](https://github.com/apache/iceberg-rust/issues/2034));
Python and Go have started.
### User Experience
**How users will use this feature:**
Build:
```cmake
# Bundle library with encryption (ON by default; requires OpenSSL)
-DICEBERG_ENCRYPTION=ON
# Optional: AWS KMS, Google Cloud KMS and Azure Key Vault clients
-DICEBERG_BUILD_KMS=ON
```
Configuration (same property names as Java):
| Level | Property | Description |
|---|---|---|
| Catalog | `encryption.kms-type` | Built-in KMS client: `aws`, `gcp` or
`azure` |
| Catalog | `encryption.kms-impl` | Name of a KMS client registered with
`KmsRegistry::Register` |
| Table | `encryption.key-id` | ID of the table master key in the KMS;
enables encryption |
| Table | `encryption.data-key-length` | Length of data keys: 16 (default),
24 or 32 bytes |
Usage example:
```cpp
// Built-in client from the (client-side) catalog properties, e.g. for REST:
// encryption.kms-type = aws
// client.region = us-west-2
auto catalog = iceberg::rest::RestCatalog::Make(config);
// Or a custom KMS client
class MyKms : public iceberg::KeyManagementClient { /* WrapKey / UnwrapKey
*/ };
iceberg::KmsRegistry::Register("my-kms", [] { return
std::make_unique<MyKms>(); });
// Tables loaded from the catalog get an EncryptingFileIO; scans, writers and
// commits encrypt and decrypt transparently. No API changes for
readers/writers.
```
Read/write behavior:
- Scans of encrypted tables decrypt manifest lists, manifests, data files,
delete files and deletion vectors
- Commits write encrypted files and add the new KEK and manifest list key
entries to `encryption-keys`
- A catalog without a KMS client can load an encrypted table but fails
cleanly on read or commit
- A wrong key fails with an error, not an exception or garbage data
### Known Limitations (not supported)
These follow the Java implementation, which does not support them yet either:
- **Column-level encryption** (per-column keys) —
[apache/iceberg#14768](https://github.com/apache/iceberg/issues/14768)
- **Encrypted table and partition statistics files** —
[apache/iceberg#17533](https://github.com/apache/iceberg/pull/17533)
- **Removing unused keys when snapshots expire** —
[apache/iceberg#16352](https://github.com/apache/iceberg/issues/16352)
- **Re-wrapping keys with a new master key** —
[apache/iceberg#18240](https://github.com/apache/iceberg/issues/18240)
- **REST KMS credential vending** —
[apache/iceberg#17155](https://github.com/apache/iceberg/pull/17155)
- **Hive catalog**: table operations are not implemented in iceberg-cpp yet,
so there is nothing to wire
- **Cloud credential providers**: the KMS clients take explicit credentials
or tokens; instance profiles, Google application default credentials and Azure
managed identities are not supported (register a custom client instead)
API Changes:
New public APIs:
```cpp
// Snapshot
std::optional<std::string> Snapshot::key_id; // JSON "key-id"
// Key management
class KeyManagementClient {
virtual Result<std::vector<uint8_t>> WrapKey(std::span<const uint8_t> key,
std::string_view
wrapping_key_id) = 0;
virtual Result<std::vector<uint8_t>> UnwrapKey(std::span<const uint8_t>
wrapped_key,
std::string_view
wrapping_key_id) = 0;
};
class KmsRegistry; // Register / Load / FromCatalogProperties
// Encryption
class EncryptionManager; // StandardEncryptionManager in the bundle
library
class EncryptionManagerRegistry;
class EncryptingFileIO; // FileIO wrapper used by tables
struct StandardKeyMetadata;
// Reader/writer options
ReaderOptions::key_metadata;
WriterOptions::key_metadata;
```
### Spec and Compatibility
- Does this change public interfaces/APIs? **Yes** — new classes above,
`Snapshot::key_id`, `ReaderOptions`/`WriterOptions::key_metadata`
- Does this change storage format? **No** — it implements the existing v3
spec fields (`encryption-keys`, snapshot `key-id`, `key_metadata`) and the AGS1
spec, byte-compatible with Java
- Interoperability is verified both ways with Java:
- C++ reads a Java-written encrypted table (AGS1 manifest lists and
manifests, PARE Parquet, AGS1 Avro data files, an encrypted deletion vector)
- Java reads a C++-written encrypted table and decrypts its deletion vector
- Small Java-generated fixtures (key metadata, AGS1 vectors, a wrapped KEK,
a small encrypted table) are checked in as test resources
### Task Breakdown
Proposed as a stacked PR series, following the order of the iceberg-rust
series. A working implementation of every step is on
[shangxinli/iceberg-cpp](https://github.com/shangxinli/iceberg-cpp/branches)
(branches `enc/01` … `enc/17`, each built on the previous one, all passing CI
locally).
**Phase 1: Foundations (no behavior change)**
- [ ] 1. Build: Parquet modular encryption and OpenSSL in the vendored Arrow
(plus zstd) — `enc/01-build`
- [ ] 2. `Snapshot::key_id` and JSON `key-id` — `enc/02-snapshot-key-id`
- [ ] 3. `StandardKeyMetadata` with Java test vectors — `enc/03-key-metadata`
- [ ] 4. AES-GCM cipher and AGS1 input/output files — `enc/04-aes-gcm-stream`
- [ ] 5. `KeyManagementClient`, `KmsRegistry`, `InMemoryKms` (tests only) —
`enc/05-kms`
- [ ] 6. `EncryptionManager` and `StandardEncryptionManager` (KEK rotation,
1-hour unwrap cache) — `enc/06-encryption-manager`
- [ ] 7. `EncryptingFileIO` — `enc/07-encrypting-file-io`
**Phase 2: Read path**
- [ ] 8. Read encrypted manifest lists — `enc/08-read-manifest-list`
- [ ] 9. Read encrypted manifests — `enc/09-read-manifest`
- [ ] 10. Read encrypted Parquet/Avro data and delete files —
`enc/10-read-data-files`
**Phase 3: Write path**
- [ ] 11. Block writes to encrypted tables until supported (no silent
plaintext) — `enc/11-block-writes`
- [ ] 12. KMS from catalog properties; in-memory, SQL and REST catalog
wiring with a trusted key ID — `enc/12-catalog-wiring`
- [ ] 13. Write encrypted manifests and manifest lists (stored length
through the Avro writer) — `enc/13-write-manifests`
- [ ] 14. Write encrypted data and delete files — `enc/14-write-data-files`
- [ ] 15. Encrypted Puffin deletion vectors — `enc/15-puffin-dv`
- [ ] 16. Enable encrypted writes, end-to-end commit test, docs; also fixes
manifest `partition-spec` metadata to be the fields array, which Java requires
— `enc/16-enable-writes`
**Phase 4: KMS clients**
- [ ] 17. AWS KMS (SigV4), Google Cloud KMS and Azure Key Vault clients in
optional `iceberg_kms` — `enc/17-kms-clients`
Feedback welcome on the library layering, the registry approach and the PR
split before the PRs are opened.
### Related Issues
- [#959](https://github.com/apache/iceberg-cpp/issues/959) — Track remaining
Iceberg v3 feature support (table-level encryption metadata)
-
[apache/iceberg-rust#2034](https://github.com/apache/iceberg-rust/issues/2034)
— Support client-side encryption (iceberg-rust epic)
-
[apache/iceberg-rust#3236](https://github.com/apache/iceberg-rust/pull/3236) —
File length in key metadata for Java interop
- [apache/iceberg#13225](https://github.com/apache/iceberg/pull/13225) —
REST catalog encryption support (Java)
- [apache/iceberg#16527](https://github.com/apache/iceberg/pull/16527) — Key
metadata format in the spec
- [apache/iceberg#16157](https://github.com/apache/iceberg/issues/16157) —
Reading encrypted deletion vectors (Java)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
