Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/15172
To give you some pointers to what I think the AES path should look like:
- decryption should work similarly to how `CryptoInputStream` works. It
uses the `Input` interface which has a `ReadableByteChannel` implementation,
but you don't really have a `ReadableByteChannel` here, just byte buffers. So
you're gonna need some code to make those work together.
- encryption: `CryptoOutputStream` has `int write(ByteBuffer src)` which
should make it easy to implement the custom `FileRegion` used in Spark.
In both cases, there might be enhancements needed to the Commons Crypto
library. That's something that we knew from the start. We should not provide a
hacky implementation in Spark just because the features don't exist yet in
Commons Crypto; this code has strict backwards compatibility requirements
because it's used in the shuffle service, so we need a stable implementation
from the start to avoid having to build version negotiation into the protocol.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
