EdwardShen125 opened a new pull request, #1567: URL: https://github.com/apache/arrow-go/pull/1567
### Rationale for this change Follow-up to the promise in #1478: *"The write side (Go producing PKMT1 key material via a CryptoFactory) is a natural follow-up; this PR is read-only."* With the read side merged, writing KMS-wrapped files still means hand-generating DEKs and hand-serializing PKMT1 key material; this adds the factory so arrow-go produces files that pyarrow's (and our own) CryptoFactory can read back (#566 tracks the umbrella discussion). ### What changes are included in this PR? The write side of `parquet/encryption`: - `NewCryptoFactory(clientFactory KmsClientFactory)` returns a factory that builds a `KmsClient` per file from a `KmsConnectionConfig` (the named type mirrors parquet-cpp's) - `EncryptionConfiguration` selects the footer master key, per-master-key column keys (each listed column gets a fresh DEK; unlisted columns stay in plaintext; empty map means uniform encryption with the footer key), single vs double wrapping (double is the parquet-cpp/pyarrow default), AAD prefix handling, and plaintext footer. Invalid configurations are rejected, including a column listed under two master keys and `DisableAadPrefixStorage` without an `AadPrefix` - `FileEncryptionProperties` generates 16-byte DEKs, wraps them (single: straight through the KMS; double: a per-(KMS instance, master key) KEK wraps DEKs locally with AES-GCM, `nonce || ciphertext || tag`, AAD = KEK ID bytes, the mirror of `decryptKeyLocally`, and only the KEK goes through the KMS), serializes the PKMT1 material, and returns `parquet.FileEncryptionProperties` for `parquet.WithEncryptionProperties` - in double wrapping mode the factory caches KEKs across files (one per master key, held for the factory's lifetime, like parquet-cpp), so writing many files costs one KMS wrap per master key — the pyarrow behavior, verified empirically: two files written by one pyarrow CryptoFactory share the footer key's KEK - `KeyMaterial` marshals `kmsInstanceID`/`kmsInstanceURL` and the KEK fields with `omitempty`, so Go-written material matches parquet-cpp's structure exactly: column keys carry no KMS instance fields, single-wrapped keys no KEK fields, field order preserved No writer changes: the existing footer and column key metadata plumbing carries the material. The interop harness gains a `read-kms` mode (pyarrow's CryptoFactory resolves keys), sharing the test KMS client with `gen-kms`. ### Are these changes tested? Unit tests cover material shape per mode, retriever round-trips, KEK sharing and freshness, per-column DEKs, uniform fallback, and the configuration errors, including KMS wrap-call counts (one per master key in double wrapping, one per key slot in single; the reuse assertion fails with the cache disabled). A Go write → Go read round trip resolves every key through `KmsKeyRetriever` alone and asserts the unwrap counts (2 / 3 for two master keys and three key slots). Against pyarrow 26.0.1 locally, with `-race`, `PARQUET_TEST_REQUIRE_PYARROW=1`, on both Go 1.26 and 1.27: Go-written files (double, single, plaintext footer) read back through pyarrow's CryptoFactory, the read-side interop tests are unchanged and still pass, and a byte-level dump of Go-written key material matches pyarrow's field for field. ### Are there any user-facing changes? New exported API in the optional `parquet/encryption` package (`CryptoFactory`, `KmsClientFactory`, `KmsConnectionConfig`, `EncryptionConfiguration`). No changes to existing APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
