EdwardShen125 opened a new pull request, #1439: URL: https://github.com/apache/arrow-go/pull/1439
### Rationale for this change #566 reports encryption and decryption failures between pyarrow and arrow-go. The direct-key half no longer reproduces on main: reading pyarrow-written files in Go works for uniform AES-GCM, AES-GCM-CTR, AAD prefix and plaintext footer, and pyarrow reads an arrow-go-written file. Checked against pyarrow 26; I suspect #596/#1390/#1412 fixed this but haven't bisected. Nothing tests these paths in either direction today, so it could regress unnoticed. What still fails is KMS. pyarrow's `CryptoFactory` writes PKMT1 key material with a wrapped DEK, arrow-go has no parser or unwrapping for it, so footer decryption fails without a custom retriever. This adds the harness proposed in #566: Go tests that drive pyarrow to generate files and check reads in both directions. The KMS test is present but skipped, marked as the known gap. For that follow-up, the question is whether PKMT1 parsing and DEK unwrapping belong in the library or as an example `KeyRetriever` like the one @zeroshade posted in #596. ### What changes are included in this PR? Two files in `parquet/`. `encryption_interop_pyarrow.py` generates the encrypted files and reads a Go-written one; it is embedded in the test binary and run through `python3`. `encryption_interop_pyarrow_test.go` holds the tests. They skip when `python3` or `pyarrow` isn't installed, so the default test run is unchanged. For CI, the existing `docker-cgo-python` job already pip-installs pyarrow and `ci/scripts/cgo_python_test.sh` activates that venv. Adding `go test -run Pyarrow ./parquet/` to that script is a one-line change, and since the install is unpinned the tests would follow the current pyarrow release. I can include it here if that sounds right. ### Are these changes tested? The tests are the change. Against pyarrow 26 locally: the four direct-key configs pass, pyarrow reads the Go-written file back correctly, and the KMS test skips with the reason above. Without python3 or pyarrow, everything skips. ### Are there any user-facing changes? No, tests only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
