EdwardShen125 opened a new pull request, #1567:
URL: https://github.com/apache/arrow-go/pull/1567

   ### Rationale for this change
   
   Follow-up to the promise in #1478: *"The write side (Go producing PKMT1 key 
material via a CryptoFactory) is a natural follow-up; this PR is read-only."* 
With the read side merged, writing KMS-wrapped files still means 
hand-generating DEKs and hand-serializing PKMT1 key material; this adds the 
factory so arrow-go produces files that pyarrow's (and our own) CryptoFactory 
can read back (#566 tracks the umbrella discussion).
   
   ### What changes are included in this PR?
   
   The write side of `parquet/encryption`:
   
   - `NewCryptoFactory(clientFactory KmsClientFactory)` returns a factory that 
builds a `KmsClient` per file from a `KmsConnectionConfig` (the named type 
mirrors parquet-cpp's)
   - `EncryptionConfiguration` selects the footer master key, per-master-key 
column keys (each listed column gets a fresh DEK; unlisted columns stay in 
plaintext; empty map means uniform encryption with the footer key), single vs 
double wrapping (double is the parquet-cpp/pyarrow default), AAD prefix 
handling, and plaintext footer. Invalid configurations are rejected, including 
a column listed under two master keys and `DisableAadPrefixStorage` without an 
`AadPrefix`
   - `FileEncryptionProperties` generates 16-byte DEKs, wraps them (single: 
straight through the KMS; double: a per-(KMS instance, master key) KEK wraps 
DEKs locally with AES-GCM, `nonce || ciphertext || tag`, AAD = KEK ID bytes, 
the mirror of `decryptKeyLocally`, and only the KEK goes through the KMS), 
serializes the PKMT1 material, and returns `parquet.FileEncryptionProperties` 
for `parquet.WithEncryptionProperties`
   - in double wrapping mode the factory caches KEKs across files (one per 
master key, held for the factory's lifetime, like parquet-cpp), so writing many 
files costs one KMS wrap per master key — the pyarrow behavior, verified 
empirically: two files written by one pyarrow CryptoFactory share the footer 
key's KEK
   - `KeyMaterial` marshals `kmsInstanceID`/`kmsInstanceURL` and the KEK fields 
with `omitempty`, so Go-written material matches parquet-cpp's structure 
exactly: column keys carry no KMS instance fields, single-wrapped keys no KEK 
fields, field order preserved
   
   No writer changes: the existing footer and column key metadata plumbing 
carries the material.
   
   The interop harness gains a `read-kms` mode (pyarrow's CryptoFactory 
resolves keys), sharing the test KMS client with `gen-kms`.
   
   ### Are these changes tested?
   
   Unit tests cover material shape per mode, retriever round-trips, KEK sharing 
and freshness, per-column DEKs, uniform fallback, and the configuration errors, 
including KMS wrap-call counts (one per master key in double wrapping, one per 
key slot in single; the reuse assertion fails with the cache disabled). A Go 
write → Go read round trip resolves every key through `KmsKeyRetriever` alone 
and asserts the unwrap counts (2 / 3 for two master keys and three key slots). 
Against pyarrow 26.0.1 locally, with `-race`, `PARQUET_TEST_REQUIRE_PYARROW=1`, 
on both Go 1.26 and 1.27: Go-written files (double, single, plaintext footer) 
read back through pyarrow's CryptoFactory, the read-side interop tests are 
unchanged and still pass, and a byte-level dump of Go-written key material 
matches pyarrow's field for field.
   
   ### Are there any user-facing changes?
   
   New exported API in the optional `parquet/encryption` package 
(`CryptoFactory`, `KmsClientFactory`, `KmsConnectionConfig`, 
`EncryptionConfiguration`). No changes to existing APIs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to