andygrove opened a new issue, #4558: URL: https://github.com/apache/datafusion-comet/issues/4558
## Background The AES functions `aes_encrypt`, `aes_decrypt`, and `try_aes_decrypt` currently fall back to Spark. They are `RuntimeReplaceable` and lower to a `StaticInvoke` of `ExpressionImplUtils.aesEncrypt` / `aesDecrypt`. Comet's `StaticInvoke` serde (`spark/src/main/scala/org/apache/comet/serde/statics.scala`) only allowlists a small set of target methods (`readSidePadding`, `isLuhnNumber`, URL `encode`/`decode`), so AES is reported as `Static invoke expression: aesEncrypt is not supported` and the query falls back. SQL file test coverage was added in #4557 (`spark/src/test/resources/sql-tests/expressions/misc/aes.sql`), currently using `query spark_answer_only` because the functions fall back. ## Proposal Add native support. Two possible routes: 1. **Codegen dispatch** (likely lowest effort): route the underlying `StaticInvoke` through Comet's codegen dispatcher so Spark's own JVM AES implementation runs over the batch. This reuses Spark's exact crypto, avoiding a Rust reimplementation and its correctness/security risk. 2. **Native implementation**: add `aesEncrypt` / `aesDecrypt` to the `StaticInvoke` allowlist in `statics.scala` with a Rust-backed scalar function. Notes: - `aes_encrypt` with GCM (default) and CBC uses a random IV, so its output is nondeterministic; tests should use round-trip (`aes_decrypt(aes_encrypt(x, k), k) = x`) rather than comparing raw ciphertext. ECB is deterministic. - Supported modes/paddings should match Spark (GCM, CBC, ECB). ## Acceptance criteria - `aes_encrypt`, `aes_decrypt`, and `try_aes_decrypt` execute natively in Comet and match Spark (round-trip for nondeterministic modes). - Promote the queries in `aes.sql` (#4557) from `query spark_answer_only` to the default `query` mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
