This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new be794c001580 [SPARK-53154][SQL][PYTHON] Add `hmac` SQL function
be794c001580 is described below
commit be794c0015803b59d213cea4a0750051d13d2d06
Author: Vinod KC <[email protected]>
AuthorDate: Wed Jul 22 19:58:57 2026 +0800
[SPARK-53154][SQL][PYTHON] Add `hmac` SQL function
### What changes were proposed in this pull request?
Add a built-in `hmac(key, message[, algorithm])` scalar function that
computes a keyed-hash message authentication code (HMAC) using the JVM's
`javax.crypto.Mac`.
**API surface added:**
- SQL: `hmac(key, message)` / `hmac(key, message, algorithm)`
- Scala DataFrame: `functions.hmac(key, message)` / `functions.hmac(key,
message, algorithm)`
- PySpark (classic + Connect): `pyspark.sql.functions.hmac(key, message,
algorithm=None)`
**Key design choices:**
- Returns `BinaryType` (raw MAC bytes), not a hex string — so the output of
one
`hmac` call feeds directly into the `key` of the next with no `unhex` in
between.
This is the critical property for chained key-derivation (e.g. AWS SigV4).
- Argument order is `(key, message)`, matching Python's `hmac.new(key,
msg)` and RFC 2104's `HMAC(K, m)`.
- Default algorithm is SHA-256. Supported: SHA-224, SHA-256, SHA-384,
SHA-512, SHA-1, MD5 (case-insensitive; both `SHA-256` and `SHA256` spellings
accepted).
### Why are the changes needed?
Spark had no single expression node for HMAC. Users had to hand-build the
RFC 2104
formula from `sha2`, `concat`, bitwise `xor`, and `unhex` primitives. The
motivating
real-world use case derives an AWS SigV4 signing key by chaining **four**
HMACs, each step's output becoming
the next step's key:
```sql
-- Four-step SigV4 key derivation (AWS docs, "Signature Version 4").
-- Each hmac() output is BinaryType, feeding directly as the next key -- no
unhex anywhere.
SELECT hex(
hmac(
hmac(
hmac(
hmac(concat('AWS4', secret), date),
region),
service),
'aws4_request')
) AS signing_key;
```
Without this function, each HMAC step is a deep sha2/unhex/concat/xor
subtree.
Nesting four of them multiplies the tree size to the point where the
Catalyst analyzer/optimizer runs out of memory on realistic inputs. hmac
collapses each level to a single StaticInvoke node.
### Does this PR introduce _any_ user-facing change?
Yes. Adds a new built-in SQL function `hmac` and corresponding Scala/PySpark
DataFrame API entries. No existing behavior is changed.
Example:
```
-- Default algorithm (SHA-256)
SELECT hex(hmac('key', 'message'));
-- 6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A
```
```
-- Explicit algorithm
SELECT hex(hmac('key', 'message', 'SHA-512'));
--
E477384D7CA229DD1426E64B63EBF2D36EBD6D7E669A6735424E72EA6C01D3F8B56EB39C36D8232F5427999B8D1A3F9CD1128FC69F4D75B434216810FA367E98
```
```
-- Chaining: binary output feeds directly as the next key
SELECT hex(hmac(hmac('key', 'message'), 'message2'));
-- 28042756E0362D954B61661BB4DEC9FF9F6C970D346CAEB6DB36ED7B735AB38C
```
### How was this patch tested?
Added tests in `ExpressionImplUtilsSuite`, `MiscExpressionsSuite` ,
`misc-functions.sql`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (sonnet-5)
Closes #57344 from vinodkc/br_SPARK-53154.
Authored-by: Vinod KC <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 2bb0b352b10584be06c3f6c5aeffe5714716a539)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 10 +++
.../source/reference/pyspark.sql/functions.rst | 1 +
python/pyspark/sql/connect/functions/builtin.py | 14 ++++
python/pyspark/sql/functions/__init__.py | 1 +
python/pyspark/sql/functions/builtin.py | 59 ++++++++++++++++
.../scala/org/apache/spark/sql/functions.scala | 34 +++++++++
.../catalyst/expressions/ExpressionImplUtils.java | 37 ++++++++++
.../sql/catalyst/analysis/FunctionRegistry.scala | 1 +
.../spark/sql/catalyst/expressions/misc.scala | 53 ++++++++++++++
.../spark/sql/errors/QueryExecutionErrors.scala | 18 +++++
.../expressions/ExpressionImplUtilsSuite.scala | 59 ++++++++++++++++
.../expressions/MiscExpressionsSuite.scala | 36 +++++++++-
.../sql-functions/sql-expression-schema.md | 1 +
.../analyzer-results/misc-functions.sql.out | 63 +++++++++++++++++
.../resources/sql-tests/inputs/misc-functions.sql | 14 ++++
.../sql-tests/results/misc-functions.sql.out | 81 ++++++++++++++++++++++
16 files changed, 481 insertions(+), 1 deletion(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 6eeff1445d07..9f6fb5ee99b2 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -4594,6 +4594,16 @@
"Invalid extension: <invalidValue>. Extension is limited to exactly
3 letters (e.g. csv, tsv, etc...)"
]
},
+ "HMAC_ALGORITHM" : {
+ "message" : [
+ "expects one of the algorithms 'SHA-224', 'SHA-256', 'SHA-384',
'SHA-512', 'SHA-1', 'MD5', but got <algorithm>."
+ ]
+ },
+ "HMAC_CRYPTO_ERROR" : {
+ "message" : [
+ "detail message: <detailMessage>"
+ ]
+ },
"INTEGER" : {
"message" : [
"expects an integer literal, but got <invalidValue>."
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst
b/python/docs/source/reference/pyspark.sql/functions.rst
index 2e6049f37870..7ce413fde0bb 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -660,6 +660,7 @@ Misc Functions
current_path
current_schema
current_user
+ hmac
input_file_block_length
input_file_block_start
input_file_name
diff --git a/python/pyspark/sql/connect/functions/builtin.py
b/python/pyspark/sql/connect/functions/builtin.py
index be5e5a4f0c6d..acf1ee6b069f 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -5383,6 +5383,20 @@ def zeroifnull(col: "ColumnOrName") -> Column:
zeroifnull.__doc__ = pysparkfuncs.zeroifnull.__doc__
+def hmac(
+ key: "ColumnOrName",
+ message: "ColumnOrName",
+ algorithm: Optional["ColumnOrName"] = None,
+) -> Column:
+ if algorithm is None:
+ return _invoke_function_over_columns("hmac", key, message)
+ else:
+ return _invoke_function_over_columns("hmac", key, message, algorithm)
+
+
+hmac.__doc__ = pysparkfuncs.hmac.__doc__
+
+
def aes_encrypt(
input: "ColumnOrName",
key: "ColumnOrName",
diff --git a/python/pyspark/sql/functions/__init__.py
b/python/pyspark/sql/functions/__init__.py
index f1a0e1de5b75..7780bae38ee9 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -520,6 +520,7 @@ __all__ = [ # noqa: F405
"current_path",
"current_schema",
"current_user",
+ "hmac",
"input_file_block_length",
"input_file_block_start",
"input_file_name",
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index 9e092ef1c4fb..77ec5353e91c 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -31491,6 +31491,65 @@ def zeroifnull(col: "ColumnOrName") -> Column:
return _invoke_function_over_columns("zeroifnull", col)
+@_try_remote_functions
+def hmac(
+ key: "ColumnOrName",
+ message: "ColumnOrName",
+ algorithm: Optional["ColumnOrName"] = None,
+) -> Column:
+ """
+ Returns the keyed-hash message authentication code (HMAC) of `message`
using `key` and the
+ given hash `algorithm`. The result is returned as raw MAC bytes; wrap it
with :func:`hex` or
+ :func:`base64` for a textual value. The default algorithm is 'SHA-256'.
+
+ .. versionadded:: 4.3.0
+
+ Parameters
+ ----------
+ key : :class:`~pyspark.sql.Column` or column name
+ The secret key, as a binary value.
+ message : :class:`~pyspark.sql.Column` or column name
+ The message to authenticate, as a binary value.
+ algorithm : :class:`~pyspark.sql.Column` or column name, optional
+ The hash algorithm. Valid values: SHA-224, SHA-256, SHA-384, SHA-512,
SHA-1, MD5.
+ The default is SHA-256.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ A new column that contains the raw HMAC bytes.
+
+ Examples
+ --------
+
+ Example 1: Compute the HMAC with the default SHA-256 algorithm.
+
+ >>> import pyspark.sql.functions as sf
+ >>> df = spark.createDataFrame([("key", "message")], ["key", "message"])
+ >>> df.select(sf.hex(sf.hmac(df.key, df.message))).show(truncate=False)
+ +----------------------------------------------------------------+
+ |hex(hmac(key, message, SHA-256)) |
+ +----------------------------------------------------------------+
+ |6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A|
+ +----------------------------------------------------------------+
+
+ Example 2: Compute the HMAC with an explicit algorithm.
+
+ >>> import pyspark.sql.functions as sf
+ >>> df = spark.createDataFrame([("key", "message")], ["key", "message"])
+ >>> df.select(sf.hex(sf.hmac(df.key, df.message,
sf.lit("SHA-1")))).show(truncate=False)
+ +----------------------------------------+
+ |hex(hmac(key, message, SHA-1)) |
+ +----------------------------------------+
+ |2088DF74D5F2146B48146CAF4965377E9D0BE3A4|
+ +----------------------------------------+
+ """
+ if algorithm is None:
+ return _invoke_function_over_columns("hmac", key, message)
+ else:
+ return _invoke_function_over_columns("hmac", key, message, algorithm)
+
+
@_try_remote_functions
def aes_encrypt(
input: "ColumnOrName",
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
index 6d824a675223..938ac3d38153 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
@@ -6633,6 +6633,40 @@ object functions {
*/
def uuid(seed: Column): Column = Column.fn("uuid", seed)
+ /**
+ * Returns the keyed-hash message authentication code (HMAC) of `message`
using `key` and the
+ * given hash `algorithm`. The result is returned as raw MAC bytes; wrap it
with `hex` or
+ * `base64` for a textual value.
+ *
+ * @param key
+ * The secret key, as a binary value.
+ * @param message
+ * The message to authenticate, as a binary value.
+ * @param algorithm
+ * The hash algorithm. Valid values: SHA-224, SHA-256, SHA-384, SHA-512,
SHA-1, MD5.
+ *
+ * @group misc_funcs
+ * @since 4.3.0
+ */
+ def hmac(key: Column, message: Column, algorithm: Column): Column =
+ Column.fn("hmac", key, message, algorithm)
+
+ /**
+ * Returns the keyed-hash message authentication code (HMAC) of `message`
using `key` and
+ * SHA-256. The result is returned as raw MAC bytes; wrap it with `hex` or
`base64` for a
+ * textual value. To use a different algorithm, call the three-argument
overload.
+ *
+ * @param key
+ * The secret key, as a binary value.
+ * @param message
+ * The message to authenticate, as a binary value.
+ *
+ * @group misc_funcs
+ * @since 4.3.0
+ */
+ def hmac(key: Column, message: Column): Column =
+ Column.fn("hmac", key, message)
+
/**
* Returns an encrypted value of `input` using AES in given `mode` with the
specified `padding`.
* Key lengths of 16, 24 and 32 bits are supported. Supported combinations
of (`mode`,
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
index e744ae3e2fc2..3f2c7e1fc5d4 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
@@ -30,6 +30,7 @@ import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
import java.util.zip.CRC32;
import javax.crypto.Cipher;
+import javax.crypto.Mac;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
@@ -181,6 +182,42 @@ public class ExpressionImplUtils {
);
}
+ /**
+ * Computes a keyed-hash message authentication code (HMAC) of the given
message using the
+ * given key and hash algorithm.
+ * @param key The secret key.
+ * @param message The message to authenticate.
+ * @param algorithm The hash algorithm. Supported values (case-insensitive):
+ * SHA-224, SHA-256, SHA-384, SHA-512, SHA-1, MD5.
+ * @return The raw HMAC bytes.
+ */
+ public static byte[] hmac(byte[] key, byte[] message, UTF8String algorithm) {
+ String macName = hmacName(algorithm.toString());
+ try {
+ Mac mac = Mac.getInstance(macName);
+ mac.init(new SecretKeySpec(key, macName));
+ return mac.doFinal(message);
+ } catch (GeneralSecurityException | IllegalArgumentException e) {
+ throw QueryExecutionErrors.hmacCryptoError(e.getMessage());
+ }
+ }
+
+ /**
+ * Maps a user-facing hash algorithm name to its JCA {@link Mac} algorithm
name. Supported
+ * algorithms are SHA-224, SHA-256, SHA-384, SHA-512, SHA-1 and MD5
(case-insensitive).
+ */
+ private static String hmacName(String algorithm) {
+ return switch (algorithm.toUpperCase(Locale.ROOT)) {
+ case "SHA-224", "SHA224" -> "HmacSHA224";
+ case "SHA-256", "SHA256" -> "HmacSHA256";
+ case "SHA-384", "SHA384" -> "HmacSHA384";
+ case "SHA-512", "SHA512" -> "HmacSHA512";
+ case "SHA-1", "SHA1" -> "HmacSHA1";
+ case "MD5" -> "HmacMD5";
+ default -> throw
QueryExecutionErrors.hmacUnsupportedAlgorithmError(algorithm);
+ };
+ }
+
/**
* Function to return the Spark version.
* @return
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index b34a5b7ad1b0..5b39aa06f1e4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -851,6 +851,7 @@ object FunctionRegistry {
expression[Sha2]("sha2"),
expression[AesEncrypt]("aes_encrypt"),
expression[AesDecrypt]("aes_decrypt"),
+ expression[Hmac]("hmac"),
expression[SparkPartitionID]("spark_partition_id"),
expression[InputFileName]("input_file_name"),
expression[InputFileBlockStart]("input_file_block_start"),
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
index 651b1a231279..e80277666109 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
@@ -636,3 +636,56 @@ case class TryAesDecrypt(
this.copy(replacement = newChild)
}
// scalastyle:on line.size.limit
+
+/**
+ * A function that computes a keyed-hash message authentication code (HMAC) of
the `message`
+ * using the given `key` and hash `algorithm`. The result is returned as raw
MAC bytes, which
+ * can be fed back in as the `key` of a subsequent HMAC to chain derivations,
or wrapped with
+ * `hex`/`base64` for a textual representation.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(key, message[, algorithm]) - Returns the keyed-hash message
authentication code (HMAC) of `message` using `key` and the hash `algorithm`.
The result is returned as raw MAC bytes; wrap it with `hex` or `base64` for a
textual value. The default algorithm is 'SHA-256'.
+ """,
+ arguments = """
+ Arguments:
+ * key - The secret key, as a binary value.
+ * message - The message to authenticate, as a binary value.
+ * algorithm - Optional hash algorithm. Valid values: SHA-224, SHA-256,
SHA-384, SHA-512, SHA-1, MD5. The default is SHA-256.
+ """,
+ examples = """
+ Examples:
+ > SELECT hex(_FUNC_('key', 'message'));
+ 6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A
+ > SELECT hex(_FUNC_('key', 'message', 'SHA-512'));
+
E477384D7CA229DD1426E64B63EBF2D36EBD6D7E669A6735424E72EA6C01D3F8B56EB39C36D8232F5427999B8D1A3F9CD1128FC69F4D75B434216810FA367E98
+ """,
+ since = "4.3.0",
+ group = "misc_funcs")
+// scalastyle:on line.size.limit
+case class Hmac(key: Expression, message: Expression, algorithm: Expression)
+ extends RuntimeReplaceable with ImplicitCastInputTypes {
+
+ override lazy val replacement: Expression = StaticInvoke(
+ classOf[ExpressionImplUtils],
+ BinaryType,
+ "hmac",
+ Seq(key, message, algorithm),
+ inputTypes)
+
+ def this(key: Expression, message: Expression) =
+ this(key, message, Literal("SHA-256"))
+
+ override def prettyName: String = "hmac"
+
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(BinaryType, BinaryType, StringTypeWithCollation(supportsTrimCollation
= true))
+
+ override def children: Seq[Expression] = Seq(key, message, algorithm)
+
+ override protected def withNewChildrenInternal(
+ newChildren: IndexedSeq[Expression]): Expression = {
+ copy(newChildren(0), newChildren(1), newChildren(2))
+ }
+}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index b54a939eb696..80feddecff30 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -2514,6 +2514,24 @@ private[sql] object QueryExecutionErrors extends
QueryErrorsBase with ExecutionE
"functionName" -> toSQLId("aes_encrypt")))
}
+ def hmacUnsupportedAlgorithmError(algorithm: String): RuntimeException = {
+ new SparkRuntimeException(
+ errorClass = "INVALID_PARAMETER_VALUE.HMAC_ALGORITHM",
+ messageParameters = Map(
+ "parameter" -> toSQLId("algorithm"),
+ "functionName" -> toSQLId("hmac"),
+ "algorithm" -> toSQLValue(algorithm, StringType)))
+ }
+
+ def hmacCryptoError(detailMessage: String): RuntimeException = {
+ new SparkRuntimeException(
+ errorClass = "INVALID_PARAMETER_VALUE.HMAC_CRYPTO_ERROR",
+ messageParameters = Map(
+ "parameter" -> toSQLId("key"),
+ "functionName" -> toSQLId("hmac"),
+ "detailMessage" -> detailMessage))
+ }
+
def hiveTableWithAnsiIntervalsError(
table: TableIdentifier): SparkUnsupportedOperationException = {
new SparkUnsupportedOperationException(
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
index e197c71026eb..e3a9177bfdbc 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala
@@ -403,6 +403,65 @@ class ExpressionImplUtilsSuite extends SparkFunSuite {
// scalastyle:on nonascii
}
+ // RFC 4231 test case 1 (HMAC-SHA-224/256/384/512) and RFC 2202 test case 1
+ // (HMAC-SHA-1 / HMAC-MD5): key is 0x0b repeated, data is "Hi There".
+ private val hmacKey20 = Array.fill[Byte](20)(0x0b)
+ private val hmacKey16 = Array.fill[Byte](16)(0x0b)
+ private val hmacData = "Hi There".getBytes("UTF-8")
+
+ private def hmacHex(key: Array[Byte], message: Array[Byte], algorithm:
String): String = {
+ ExpressionImplUtils.hmac(key, message, UTF8String.fromString(algorithm))
+ .map(b => f"$b%02x").mkString
+ }
+
+ test("Hmac with supported algorithms") {
+ assert(hmacHex(hmacKey20, hmacData, "SHA-224") ==
+ "896fb1128abbdf196832107cd49df33f47b4b1169912ba4f53684b22")
+ assert(hmacHex(hmacKey20, hmacData, "SHA-256") ==
+ "b0344c61d8db38535ca8afceaf0bf12b881dc200c9833da726e9376c2e32cff7")
+ assert(hmacHex(hmacKey20, hmacData, "SHA-384") ==
+ "afd03944d84895626b0825f4ab46907f15f9dadbe4101ec682aa034c7cebc59c" +
+ "faea9ea9076ede7f4af152e8b2fa9cb6")
+ assert(hmacHex(hmacKey20, hmacData, "SHA-512") ==
+ "87aa7cdea5ef619d4ff0b4241a1d6cb02379f4e2ce4ec2787ad0b30545e17cde" +
+ "daa833b7d6b8a702038b274eaea3f4e4be9d914eeb61f1702e696c203a126854")
+ assert(hmacHex(hmacKey20, hmacData, "SHA-1") ==
+ "b617318655057264e28bc0b6fb378c8ef146be00")
+ assert(hmacHex(hmacKey16, hmacData, "MD5") ==
+ "9294727a3638bb1c13f48ef8158bfc9d")
+ }
+
+ test("Hmac algorithm name is case-insensitive and accepts the compact form")
{
+ val expected =
"b0344c61d8db38535ca8afceaf0bf12b881dc200c9833da726e9376c2e32cff7"
+ assert(hmacHex(hmacKey20, hmacData, "sha-256") == expected)
+ assert(hmacHex(hmacKey20, hmacData, "SHA256") == expected)
+ }
+
+ test("Hmac unsupported algorithm error") {
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ ExpressionImplUtils.hmac(hmacKey20, hmacData,
UTF8String.fromString("SHA-3"))
+ },
+ condition = "INVALID_PARAMETER_VALUE.HMAC_ALGORITHM",
+ parameters = Map(
+ "parameter" -> "`algorithm`",
+ "functionName" -> "`hmac`",
+ "algorithm" -> "'SHA-3'"))
+ }
+
+ test("Hmac empty key error") {
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ ExpressionImplUtils.hmac(Array.empty[Byte], hmacData,
UTF8String.fromString("SHA-256"))
+ },
+ condition = "INVALID_PARAMETER_VALUE.HMAC_CRYPTO_ERROR",
+ parameters = Map(
+ "parameter" -> "`key`",
+ "functionName" -> "`hmac`",
+ "detailMessage" -> ".*"),
+ matchPVals = true)
+ }
+
test("TryValidate UTF8 string") {
def tryValidateUTF8(str: UTF8String, expected: UTF8String): Unit = {
assert(ExpressionImplUtils.tryValidateUTF8String(str) == expected)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
index ff0183eb265b..032762470900 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
@@ -21,7 +21,7 @@ import java.io.PrintStream
import scala.util.Random
-import org.apache.spark.SparkFunSuite
+import org.apache.spark.{SparkFunSuite, SparkRuntimeException}
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
import org.apache.spark.sql.catalyst.util.TypeUtils.ordinalNumber
import org.apache.spark.sql.types._
@@ -112,4 +112,38 @@ class MiscExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(outputCodegen.contains(s"Result of $inputExpr is 1"))
assert(outputEval.contains(s"Result of $inputExpr is 1"))
}
+
+ test("Hmac") {
+ def bytes(hex: String): Array[Byte] =
+ hex.grouped(2).map(Integer.parseInt(_, 16).toByte).toArray
+
+ val key = Literal("key".getBytes("UTF-8"))
+ val message = Literal("message".getBytes("UTF-8"))
+
+ // Default algorithm is SHA-256.
+ checkEvaluation(
+ new Hmac(key, message),
+
bytes("6e9ef29b75fffc5b7abae527d58fdadb2fe42e7219011976917343065f58ed4a"))
+
+ // Explicit algorithm.
+ checkEvaluation(
+ Hmac(key, message, Literal("SHA-1")),
+ bytes("2088df74d5f2146b48146caf4965377e9d0be3a4"))
+
+ // Null propagation.
+ checkEvaluation(Hmac(Literal.create(null, BinaryType), message,
Literal("SHA-256")), null)
+ checkEvaluation(Hmac(key, Literal.create(null, BinaryType),
Literal("SHA-256")), null)
+ checkEvaluation(Hmac(key, message, Literal.create(null, StringType)), null)
+ }
+
+ test("Hmac unsupported algorithm") {
+ checkErrorInExpression[SparkRuntimeException](
+ Hmac(Literal("key".getBytes("UTF-8")),
Literal("message".getBytes("UTF-8")),
+ Literal("SHA-3")),
+ "INVALID_PARAMETER_VALUE.HMAC_ALGORITHM",
+ Map(
+ "parameter" -> "`algorithm`",
+ "functionName" -> "`hmac`",
+ "algorithm" -> "'SHA-3'"))
+ }
}
diff --git a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
index 4ed20ba4aa93..99285d126acd 100644
--- a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
+++ b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
@@ -166,6 +166,7 @@
| org.apache.spark.sql.catalyst.expressions.Hex | hex | SELECT hex(17) |
struct<hex(17):string> |
| org.apache.spark.sql.catalyst.expressions.HllSketchEstimate |
hll_sketch_estimate | SELECT hll_sketch_estimate(hll_sketch_agg(col)) FROM
VALUES (1), (1), (2), (2), (3) tab(col) |
struct<hll_sketch_estimate(hll_sketch_agg(col, 12)):bigint> |
| org.apache.spark.sql.catalyst.expressions.HllUnion | hll_union | SELECT
hll_sketch_estimate(hll_union(hll_sketch_agg(col1), hll_sketch_agg(col2))) FROM
VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6) tab(col1, col2) |
struct<hll_sketch_estimate(hll_union(hll_sketch_agg(col1, 12),
hll_sketch_agg(col2, 12), false)):bigint> |
+| org.apache.spark.sql.catalyst.expressions.Hmac | hmac | SELECT
hex(hmac('key', 'message')) | struct<hex(hmac(key, message, SHA-256)):string> |
| org.apache.spark.sql.catalyst.expressions.HourExpressionBuilder | hour |
SELECT hour('2018-02-14 12:58:59') | struct<hour(2018-02-14 12:58:59):int> |
| org.apache.spark.sql.catalyst.expressions.Hypot | hypot | SELECT hypot(3, 4)
| struct<HYPOT(3, 4):double> |
| org.apache.spark.sql.catalyst.expressions.ILike | ilike | SELECT
ilike('Spark', '_Park') | struct<ilike(Spark, _Park):boolean> |
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/misc-functions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/misc-functions.sql.out
index 708ba0ad7b58..a470d33d9893 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/misc-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/misc-functions.sql.out
@@ -233,3 +233,66 @@ SELECT assert_true(col1 <= col2) FROM VALUES
('2025-03-01', '2025-03-10')
-- !query analysis
Project [assert_true((col1#x <= col2#x), '(col1 <= col2)' is not true!) AS
assert_true((col1 <= col2), '(col1 <= col2)' is not true!)#x]
+- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+SELECT hex(hmac('key', 'message'))
+-- !query analysis
+Project [hex(hmac(cast(key as binary), cast(message as binary), SHA-256)) AS
hex(hmac(key, message, SHA-256))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hex(hmac('key', 'message', 'SHA-256'))
+-- !query analysis
+Project [hex(hmac(cast(key as binary), cast(message as binary), SHA-256)) AS
hex(hmac(key, message, SHA-256))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hex(hmac('key', 'message', 'SHA-512'))
+-- !query analysis
+Project [hex(hmac(cast(key as binary), cast(message as binary), SHA-512)) AS
hex(hmac(key, message, SHA-512))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT lower(hex(hmac('key', 'message', 'sha256')))
+-- !query analysis
+Project [lower(hex(hmac(cast(key as binary), cast(message as binary),
sha256))) AS lower(hex(hmac(key, message, sha256)))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hex(hmac(hmac('key', 'message'), 'message2'))
+-- !query analysis
+Project [hex(hmac(hmac(cast(key as binary), cast(message as binary), SHA-256),
cast(message2 as binary), SHA-256)) AS hex(hmac(hmac(key, message, SHA-256),
message2, SHA-256))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hmac(CAST(NULL AS BINARY), 'message')
+-- !query analysis
+Project [hmac(cast(null as binary), cast(message as binary), SHA-256) AS
hmac(CAST(NULL AS BINARY), message, SHA-256)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hmac('key', CAST(NULL AS BINARY))
+-- !query analysis
+Project [hmac(cast(key as binary), cast(null as binary), SHA-256) AS hmac(key,
CAST(NULL AS BINARY), SHA-256)#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hmac('key', 'message', CAST(NULL AS STRING))
+-- !query analysis
+Project [hmac(cast(key as binary), cast(message as binary), cast(null as
string)) AS hmac(key, message, CAST(NULL AS STRING))#x]
++- OneRowRelation
+
+
+-- !query
+SELECT hmac('key', 'message', 'SHA-3')
+-- !query analysis
+Project [hmac(cast(key as binary), cast(message as binary), SHA-3) AS
hmac(key, message, SHA-3)#x]
++- OneRowRelation
diff --git a/sql/core/src/test/resources/sql-tests/inputs/misc-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/misc-functions.sql
index 417af4220641..e0b7b6ac3f88 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/misc-functions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/misc-functions.sql
@@ -44,3 +44,17 @@ SET spark.sql.legacy.raiseErrorWithoutErrorClass=false;
-- Implicit alias of assert expression
SELECT assert_true(col1 <= col2) FROM VALUES ('2025-03-01', '2025-03-10');
+
+-- hmac
+SELECT hex(hmac('key', 'message'));
+SELECT hex(hmac('key', 'message', 'SHA-256'));
+SELECT hex(hmac('key', 'message', 'SHA-512'));
+SELECT lower(hex(hmac('key', 'message', 'sha256')));
+-- Chaining: the binary output of one hmac feeds in as the key of the next.
+SELECT hex(hmac(hmac('key', 'message'), 'message2'));
+-- Null propagation.
+SELECT hmac(CAST(NULL AS BINARY), 'message');
+SELECT hmac('key', CAST(NULL AS BINARY));
+SELECT hmac('key', 'message', CAST(NULL AS STRING));
+-- Unsupported algorithm.
+SELECT hmac('key', 'message', 'SHA-3');
diff --git
a/sql/core/src/test/resources/sql-tests/results/misc-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/misc-functions.sql.out
index efcb0409bf93..9373dc63289a 100644
--- a/sql/core/src/test/resources/sql-tests/results/misc-functions.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/misc-functions.sql.out
@@ -321,3 +321,84 @@ SELECT assert_true(col1 <= col2) FROM VALUES
('2025-03-01', '2025-03-10')
struct<assert_true((col1 <= col2), '(col1 <= col2)' is not true!):void>
-- !query output
NULL
+
+
+-- !query
+SELECT hex(hmac('key', 'message'))
+-- !query schema
+struct<hex(hmac(key, message, SHA-256)):string>
+-- !query output
+6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A
+
+
+-- !query
+SELECT hex(hmac('key', 'message', 'SHA-256'))
+-- !query schema
+struct<hex(hmac(key, message, SHA-256)):string>
+-- !query output
+6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A
+
+
+-- !query
+SELECT hex(hmac('key', 'message', 'SHA-512'))
+-- !query schema
+struct<hex(hmac(key, message, SHA-512)):string>
+-- !query output
+E477384D7CA229DD1426E64B63EBF2D36EBD6D7E669A6735424E72EA6C01D3F8B56EB39C36D8232F5427999B8D1A3F9CD1128FC69F4D75B434216810FA367E98
+
+
+-- !query
+SELECT lower(hex(hmac('key', 'message', 'sha256')))
+-- !query schema
+struct<lower(hex(hmac(key, message, sha256))):string>
+-- !query output
+6e9ef29b75fffc5b7abae527d58fdadb2fe42e7219011976917343065f58ed4a
+
+
+-- !query
+SELECT hex(hmac(hmac('key', 'message'), 'message2'))
+-- !query schema
+struct<hex(hmac(hmac(key, message, SHA-256), message2, SHA-256)):string>
+-- !query output
+28042756E0362D954B61661BB4DEC9FF9F6C970D346CAEB6DB36ED7B735AB38C
+
+
+-- !query
+SELECT hmac(CAST(NULL AS BINARY), 'message')
+-- !query schema
+struct<hmac(CAST(NULL AS BINARY), message, SHA-256):binary>
+-- !query output
+NULL
+
+
+-- !query
+SELECT hmac('key', CAST(NULL AS BINARY))
+-- !query schema
+struct<hmac(key, CAST(NULL AS BINARY), SHA-256):binary>
+-- !query output
+NULL
+
+
+-- !query
+SELECT hmac('key', 'message', CAST(NULL AS STRING))
+-- !query schema
+struct<hmac(key, message, CAST(NULL AS STRING)):binary>
+-- !query output
+NULL
+
+
+-- !query
+SELECT hmac('key', 'message', 'SHA-3')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "INVALID_PARAMETER_VALUE.HMAC_ALGORITHM",
+ "sqlState" : "22023",
+ "messageParameters" : {
+ "algorithm" : "'SHA-3'",
+ "functionName" : "`hmac`",
+ "parameter" : "`algorithm`"
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
