andygrove opened a new issue, #295:
URL: https://github.com/apache/datafusion-spark/issues/295

   Part of #272 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `aes_encrypt(expr, key[, mode[, padding[, iv[, aad]]]])` encrypts `expr` 
with AES and returns `BINARY`.
   `expr`, `key`, `iv` and `aad` are `BINARY`; `mode` and `padding` are 
`STRING` (collated strings accepted from 4.0).
   Through `ImplicitCastInputTypes`, string arguments for `expr`, `key`, `iv` 
and `aad` are cast to binary (their UTF-8
   bytes). Defaults: `mode = 'GCM'`, `padding = 'DEFAULT'`, `iv = ''`, `aad = 
''`. Since 3.3.0 (`iv` and `aad` since
   3.5.0). The result is NULL if any argument is NULL (`StaticInvoke` with null 
propagation).
   
   Supported `(mode, padding)` pairs, both case-insensitive: `ECB` with 
`PKCS`/`DEFAULT` (`AES/ECB/PKCS5Padding`),
   `CBC` with `PKCS`/`DEFAULT` (`AES/CBC/PKCS5Padding`), `GCM` with 
`NONE`/`DEFAULT` (`AES/GCM/NoPadding`, 128-bit tag).
   The key must be 16, 24 or 32 bytes (AES-128/192/256). It is used as is, with 
no key derivation.
   
   Output layout: ECB is the ciphertext only. CBC is `iv (16 bytes) || 
ciphertext`. GCM is
   `iv (12 bytes) || ciphertext || tag (16 bytes)`. If `iv` is empty, a random 
IV is drawn from `SecureRandom`, so CBC and
   GCM output is nondeterministic. Only ECB, or CBC/GCM with an explicit IV, 
give fixed results.
   
   ### Examples
   
   Deterministic ones (from the `ExpressionDescription`, 
`ExpressionImplUtilsSuite` and `DataFrameFunctionsSuite`):
   
   ```sql
   SELECT base64(aes_encrypt('Spark SQL', '1234567890abcdef', 'ECB', 'PKCS'));
   -- 3lmwu+Mw0H3fi5NDvcu9lg==
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop', 'ECB'));
   -- 4Hv0UKCx6nfUeAoPZo1z+w==
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678', 'ECB'));
   -- NeTYNgA+PCQBN50DA//O2w==
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 
'ECB'));
   -- 9J3iZbIxnmaG+OIA9Amd+A==
   SELECT base64(aes_encrypt('', 'abcdefghijklmnop', 'ECB'));
   -- jmTOhz8XTbskI/zYFFgOFQ==
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 
'CBC', 'DEFAULT', unhex('00000000000000000000000000000000')));
   -- AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg=
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 
'GCM', 'DEFAULT', unhex('000000000000000000000000')));
   -- AAAAAAAAAAAAAAAAQiYi+sRNYDAOTjdSEcYBFsAWPL1f
   SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 
'GCM', 'DEFAULT', unhex('000000000000000000000000'), 'This is an AAD mixed into 
the input'));
   -- AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4
   SELECT aes_encrypt(cast(null as string), 'abcdefghijklmnop');
   -- NULL
   ```
   
   Random-IV examples can only be checked by round trip, or by length:
   
   ```sql
   SELECT cast(aes_decrypt(aes_encrypt('Spark', 'abcdefghijklmnop'), 
'abcdefghijklmnop') AS STRING);
   -- Spark
   SELECT length(aes_encrypt('Spark', '0000111122223333'));        -- 33 (12 IV 
+ 5 + 16 tag), by code reading
   SELECT length(aes_encrypt('Apache Spark', '1234567890abcdef', 'CBC'));  -- 
32 (16 IV + 16), by code reading
   ```
   
   ### Edge cases to test
   
   - NULL in each of the six arguments gives NULL. String vs binary `expr` and 
`key` give the same bytes.
   - Key length other than 16/24/32 (0, 15, 17 bytes): 
`INVALID_PARAMETER_VALUE.AES_KEY_LENGTH`, sqlState 22023,
     "The value of parameter(s) `key` in `aes_encrypt`/`aes_decrypt` is 
invalid: expects a binary value with 16, 24 or
     32 bytes, but got 17 bytes." The key is checked before the mode.
   - Unsupported pair (`'CBC','None'`, `'ECB','NoPadding'`, `'GCM','PKCS'`, 
`'GSM'`): `UNSUPPORTED_FEATURE.AES_MODE`,
     sqlState 0A000, "The feature is not supported: AES-CBC with the padding 
None by the `aes_encrypt`/`aes_decrypt`
     function." The parameters echo the user's original casing.
   - Non-empty IV with ECB: `UNSUPPORTED_FEATURE.AES_MODE_IV` (`aes_encrypt` 
with AES-ECB does not support
     initialization vectors (IVs).).
   - Wrong IV length (5 bytes for CBC or GCM): 
`INVALID_PARAMETER_VALUE.AES_IV_LENGTH` with parameter `` `iv` `` and
     "supports 16-byte CBC IVs and 12-byte GCM IVs, but got 5 bytes for CBC."
   - Non-empty AAD with ECB or CBC: `UNSUPPORTED_FEATURE.AES_MODE_AAD`. Empty 
`iv` / `aad` (`''`) behave exactly like
     the defaults, so ECB with `iv = ''` and `aad = ''` is accepted.
   - Any JCE failure becomes `INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR` with 
parameter `` `expr`, `key` `` and the JCE
     message as `detailMessage`. A Rust implementation can't reproduce those 
messages exactly; match the condition.
   - Empty plaintext: ECB and CBC produce one full padding block, GCM produces 
IV + tag only (28 bytes).
   - Plaintext longer than one block (`This message is longer than a single AES 
block and should work fine.`) and
     multi-byte UTF-8 strings.
   - Literal and column for every argument, including per-row `mode`, 
`padding`, `key` and `iv` columns.
   - Determinism: Spark does not mark `aes_encrypt` nondeterministic. A call 
where every argument is a literal is
     constant-folded (`StaticInvoke` is foldable), so every row gets the same 
IV. With a column argument each row gets a
     fresh IV. In DataFusion the UDF should be `Volatile`. Tests must 
round-trip or pass a fixed IV.
   
   ### Differences across Spark versions
   
   - 3.4: only ECB and GCM, signature `(expr, key[, mode[, padding]])`. Only 
`GeneralSecurityException` is mapped to
     `AES_CRYPTO_ERROR`.
   - 3.5: adds CBC (SPARK-43038), `iv` and `aad`, 
`UNSUPPORTED_FEATURE.AES_MODE_IV`, `UNSUPPORTED_FEATURE.AES_MODE_AAD`,
     `INVALID_PARAMETER_VALUE.AES_IV_LENGTH`. `IllegalArgumentException` from 
the JCE is also mapped to
     `AES_CRYPTO_ERROR`.
   - 4.0: `mode` / `padding` accept collated strings. No behavior change 
through 4.2.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala`
 (`AesEncrypt`, a
     `RuntimeReplaceable` over `StaticInvoke`) and
     
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java`
     (`aesEncrypt`, `aesInternal`, `CipherMode`). Errors in 
`QueryExecutionErrors.scala`.
   - DataFusion core has no AES functions.
   - Suggested crates: RustCrypto `aes`, `aes-gcm`, `cbc`, `ecb` (with 
`block-padding` PKCS7, which matches JCE
     `PKCS5Padding` for 16-byte blocks). Random IVs from `rand` (already a 
dependency) with an OS RNG.
   - Put it in `crates/spark/src/function/misc/` (Spark group `misc_funcs`), 
sharing a helper module with
     `aes_decrypt` and `try_aes_decrypt` for mode/padding parsing, key checks 
and error construction.
   - Tests: add `crates/sqllogictest/test_files/misc/aes_encrypt.slt` with the 
fixed-IV and ECB vectors above and
     round-trip tests through `aes_decrypt`; document with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to