andygrove opened a new issue, #295:
URL: https://github.com/apache/datafusion-spark/issues/295
Part of #272 and apache/datafusion-comet#7300.
### What the function does
`aes_encrypt(expr, key[, mode[, padding[, iv[, aad]]]])` encrypts `expr`
with AES and returns `BINARY`.
`expr`, `key`, `iv` and `aad` are `BINARY`; `mode` and `padding` are
`STRING` (collated strings accepted from 4.0).
Through `ImplicitCastInputTypes`, string arguments for `expr`, `key`, `iv`
and `aad` are cast to binary (their UTF-8
bytes). Defaults: `mode = 'GCM'`, `padding = 'DEFAULT'`, `iv = ''`, `aad =
''`. Since 3.3.0 (`iv` and `aad` since
3.5.0). The result is NULL if any argument is NULL (`StaticInvoke` with null
propagation).
Supported `(mode, padding)` pairs, both case-insensitive: `ECB` with
`PKCS`/`DEFAULT` (`AES/ECB/PKCS5Padding`),
`CBC` with `PKCS`/`DEFAULT` (`AES/CBC/PKCS5Padding`), `GCM` with
`NONE`/`DEFAULT` (`AES/GCM/NoPadding`, 128-bit tag).
The key must be 16, 24 or 32 bytes (AES-128/192/256). It is used as is, with
no key derivation.
Output layout: ECB is the ciphertext only. CBC is `iv (16 bytes) ||
ciphertext`. GCM is
`iv (12 bytes) || ciphertext || tag (16 bytes)`. If `iv` is empty, a random
IV is drawn from `SecureRandom`, so CBC and
GCM output is nondeterministic. Only ECB, or CBC/GCM with an explicit IV,
give fixed results.
### Examples
Deterministic ones (from the `ExpressionDescription`,
`ExpressionImplUtilsSuite` and `DataFrameFunctionsSuite`):
```sql
SELECT base64(aes_encrypt('Spark SQL', '1234567890abcdef', 'ECB', 'PKCS'));
-- 3lmwu+Mw0H3fi5NDvcu9lg==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop', 'ECB'));
-- 4Hv0UKCx6nfUeAoPZo1z+w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678', 'ECB'));
-- NeTYNgA+PCQBN50DA//O2w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH',
'ECB'));
-- 9J3iZbIxnmaG+OIA9Amd+A==
SELECT base64(aes_encrypt('', 'abcdefghijklmnop', 'ECB'));
-- jmTOhz8XTbskI/zYFFgOFQ==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH',
'CBC', 'DEFAULT', unhex('00000000000000000000000000000000')));
-- AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg=
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH',
'GCM', 'DEFAULT', unhex('000000000000000000000000')));
-- AAAAAAAAAAAAAAAAQiYi+sRNYDAOTjdSEcYBFsAWPL1f
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH',
'GCM', 'DEFAULT', unhex('000000000000000000000000'), 'This is an AAD mixed into
the input'));
-- AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4
SELECT aes_encrypt(cast(null as string), 'abcdefghijklmnop');
-- NULL
```
Random-IV examples can only be checked by round trip, or by length:
```sql
SELECT cast(aes_decrypt(aes_encrypt('Spark', 'abcdefghijklmnop'),
'abcdefghijklmnop') AS STRING);
-- Spark
SELECT length(aes_encrypt('Spark', '0000111122223333')); -- 33 (12 IV
+ 5 + 16 tag), by code reading
SELECT length(aes_encrypt('Apache Spark', '1234567890abcdef', 'CBC')); --
32 (16 IV + 16), by code reading
```
### Edge cases to test
- NULL in each of the six arguments gives NULL. String vs binary `expr` and
`key` give the same bytes.
- Key length other than 16/24/32 (0, 15, 17 bytes):
`INVALID_PARAMETER_VALUE.AES_KEY_LENGTH`, sqlState 22023,
"The value of parameter(s) `key` in `aes_encrypt`/`aes_decrypt` is
invalid: expects a binary value with 16, 24 or
32 bytes, but got 17 bytes." The key is checked before the mode.
- Unsupported pair (`'CBC','None'`, `'ECB','NoPadding'`, `'GCM','PKCS'`,
`'GSM'`): `UNSUPPORTED_FEATURE.AES_MODE`,
sqlState 0A000, "The feature is not supported: AES-CBC with the padding
None by the `aes_encrypt`/`aes_decrypt`
function." The parameters echo the user's original casing.
- Non-empty IV with ECB: `UNSUPPORTED_FEATURE.AES_MODE_IV` (`aes_encrypt`
with AES-ECB does not support
initialization vectors (IVs).).
- Wrong IV length (5 bytes for CBC or GCM):
`INVALID_PARAMETER_VALUE.AES_IV_LENGTH` with parameter `` `iv` `` and
"supports 16-byte CBC IVs and 12-byte GCM IVs, but got 5 bytes for CBC."
- Non-empty AAD with ECB or CBC: `UNSUPPORTED_FEATURE.AES_MODE_AAD`. Empty
`iv` / `aad` (`''`) behave exactly like
the defaults, so ECB with `iv = ''` and `aad = ''` is accepted.
- Any JCE failure becomes `INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR` with
parameter `` `expr`, `key` `` and the JCE
message as `detailMessage`. A Rust implementation can't reproduce those
messages exactly; match the condition.
- Empty plaintext: ECB and CBC produce one full padding block, GCM produces
IV + tag only (28 bytes).
- Plaintext longer than one block (`This message is longer than a single AES
block and should work fine.`) and
multi-byte UTF-8 strings.
- Literal and column for every argument, including per-row `mode`,
`padding`, `key` and `iv` columns.
- Determinism: Spark does not mark `aes_encrypt` nondeterministic. A call
where every argument is a literal is
constant-folded (`StaticInvoke` is foldable), so every row gets the same
IV. With a column argument each row gets a
fresh IV. In DataFusion the UDF should be `Volatile`. Tests must
round-trip or pass a fixed IV.
### Differences across Spark versions
- 3.4: only ECB and GCM, signature `(expr, key[, mode[, padding]])`. Only
`GeneralSecurityException` is mapped to
`AES_CRYPTO_ERROR`.
- 3.5: adds CBC (SPARK-43038), `iv` and `aad`,
`UNSUPPORTED_FEATURE.AES_MODE_IV`, `UNSUPPORTED_FEATURE.AES_MODE_AAD`,
`INVALID_PARAMETER_VALUE.AES_IV_LENGTH`. `IllegalArgumentException` from
the JCE is also mapped to
`AES_CRYPTO_ERROR`.
- 4.0: `mode` / `padding` accept collated strings. No behavior change
through 4.2.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala`
(`AesEncrypt`, a
`RuntimeReplaceable` over `StaticInvoke`) and
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java`
(`aesEncrypt`, `aesInternal`, `CipherMode`). Errors in
`QueryExecutionErrors.scala`.
- DataFusion core has no AES functions.
- Suggested crates: RustCrypto `aes`, `aes-gcm`, `cbc`, `ecb` (with
`block-padding` PKCS7, which matches JCE
`PKCS5Padding` for 16-byte blocks). Random IVs from `rand` (already a
dependency) with an OS RNG.
- Put it in `crates/spark/src/function/misc/` (Spark group `misc_funcs`),
sharing a helper module with
`aes_decrypt` and `try_aes_decrypt` for mode/padding parsing, key checks
and error construction.
- Tests: add `crates/sqllogictest/test_files/misc/aes_encrypt.slt` with the
fixed-IV and ECB vectors above and
round-trip tests through `aes_decrypt`; document with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]