This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d42925f18dde [SPARK-58440][SQL] Convert Sequence codegen error
parameters to String
d42925f18dde is described below
commit d42925f18dde75dc383aa62eaf6aab3432916b34
Author: YangJie <[email protected]>
AuthorDate: Fri Jul 31 20:27:27 2026 +0800
[SPARK-58440][SQL] Convert Sequence codegen error parameters to String
### What changes were proposed in this pull request?
`Sequence.genSequenceLengthCode` generates Java source that builds the
message parameters for `_LEGACY_ERROR_TEMP_3243`:
```java
java.util.Map<String, String> params = new java.util.HashMap<String,
String>();
params.put("start", $start);
params.put("stop", $stop);
params.put("step", $step);
throw new
org.apache.spark.SparkIllegalArgumentException("_LEGACY_ERROR_TEMP_3243",
params);
```
`$start` and `$stop` are numeric expressions, and `$step` is numeric or a
`CalendarInterval` reference depending on the sequence implementation. Janino,
which compiles the generated code, erases the type arguments and binds these
calls to `put(Object, Object)`, so the raw values go in unconverted. This PR
converts them with `String.valueOf(...)`.
### Why are the changes needed?
The parameter map reaches `SparkIllegalArgumentException` and is returned
by `SparkThrowable.getMessageParameters()`, whose declared type is
`java.util.Map<String, String>`. Callers that read a value as a `String` - the
declared type - fail with a `ClassCastException`. Confirmed consumers on this
path:
- `SparkThrowableHelper.scala:165` does `value.replaceAll("#\\d+", "#x")`,
used when `spark.sql.error.messageFormat` is `MINIMAL`/`STANDARD` (spark-sql
CLI and Thrift Server via `HiveThriftServerErrors.scala:43-44`).
- Spark Connect's `ErrorUtils.scala:241` renders the map through json4s
`map2jvalue`, which inserts a `checkcast String` per value, and
`ErrorUtils.scala:167-169` puts it into a proto `map<string, string>`.
- `CheckErrorHelper.checkError` itself: while building its diagnostic dump
it iterates the actual parameters as `String`, so it throws
`ClassCastException: class java.lang.Integer cannot be cast to class
java.lang.String` at `CheckErrorHelper.scala:186`.
The rendered message text was never affected, because parameter
substitution calls `toString`. That is why this went unnoticed: only the
structured parameter map was wrong, and this error path had no test coverage.
The map-based parameters were introduced by SPARK-46991 (`f5b0de07eff`,
replacing `IllegalArgumentException` with `SparkIllegalArgumentException` in
catalyst), so this is present from 4.0.0 onward. Verified on `master`,
`branch-4.x`, `branch-4.2`, `branch-4.1` and `branch-4.0`. `branch-3.5` is not
affected - it still builds the message by string concatenation.
Out of scope, noted for the record: for the same condition the interpreted
path (`Sequence.getSequenceLength`) throws a plain `IllegalArgumentException`
via `require` with no error class, so codegen and interpreted execution are not
symmetric here. Reconciling that would change a user-visible exception type and
belongs in its own change.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix. `getMessageParameters()` on this error now returns `String`
values as its signature declares, instead of
`Byte`/`Short`/`Integer`/`Long`/`CalendarInterval`. The rendered message text
is unchanged.
### How was this patch tested?
New test in `CollectionExpressionsSuite`, asserting the parameter map
rather than the message text, covering both code paths: the integral
implementation (`$step` is a Java primitive) and `TemporalSequenceImpl`
(`$step` is a `CalendarInterval`). Confirmed it fails without the fix and
passes with it. The test is codegen-only because the interpreted path has no
error class, which is also why `checkErrorInExpression` cannot be used - it
additionally runs NO_CODEGEN.
Beyond the committed test I probed the full static-type matrix under Janino
- `byte`/`short`/`int`/`long` for the integral path, plus `CalendarInterval`,
`Period`, `Duration` and `TimestampNTZ` steps - and confirmed that all nine
cases hold non-`String` values before the fix, `java.lang.String` after, and
that the rendered message is byte-for-byte identical in every case. `byte` and
`short` have no dedicated `String.valueOf` overload; Janino widens them to
`String.valueOf(int)` the sa [...]
`CollectionExpressionsSuite` (62 tests) and `dev/scalastyle` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code Opus 5
Closes #57664 from LuciferYang/SPARK-58440.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit 6a7a7f85bee02935601da135da328ed95232f77d)
Signed-off-by: yangjie01 <[email protected]>
---
.../expressions/collectionOperations.scala | 12 +++++--
.../expressions/CollectionExpressionsSuite.scala | 37 ++++++++++++++++++++--
2 files changed, 44 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index f9dabd38664a..48096408c403 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -3987,14 +3987,20 @@ object Sequence {
estimatedStep: String,
len: String): String = {
val calcFn = classOf[Sequence].getName + ".sequenceLength"
+ // `$start` and `$stop` are numeric expressions and `$step` is numeric or a
+ // CalendarInterval reference, so they have to be converted before going
into a
+ // `Map<String, String>`. Janino, which compiles the generated code,
erases the type
+ // arguments and binds `put` to `put(Object, Object)`, which lets the raw
values through
+ // and leaves the parameter map holding non-String values that
+ // `SparkThrowable.getMessageParameters` then hands out as Strings.
s"""
|if (!(($estimatedStep > 0 && $start <= $stop) ||
| ($estimatedStep < 0 && $start >= $stop) ||
| ($estimatedStep == 0 && $start == $stop))) {
| java.util.Map<String, String> params = new java.util.HashMap<String,
String>();
- | params.put("start", $start);
- | params.put("stop", $stop);
- | params.put("step", $step);
+ | params.put("start", String.valueOf($start));
+ | params.put("stop", String.valueOf($stop));
+ | params.put("step", String.valueOf($step));
| throw new org.apache.spark.SparkIllegalArgumentException(
| "_LEGACY_ERROR_TEMP_3243", params);
|}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 37e0b53dd46d..35c632ec5ad7 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -24,13 +24,13 @@ import java.util.TimeZone
import scala.language.implicitConversions
import scala.util.Random
-import org.apache.spark.{SparkArrayIndexOutOfBoundsException, SparkFunSuite,
SparkRuntimeException}
+import org.apache.spark.{SparkArrayIndexOutOfBoundsException, SparkFunSuite,
SparkIllegalArgumentException, SparkRuntimeException}
import org.apache.spark.sql.Row
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData,
DateTimeTestUtils, DateTimeUtils, GenericArrayData}
-import
org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{outstandingZoneIds, LA,
UTC}
+import
org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{outstandingZoneIds, LA,
UTC, UTC_OPT}
import org.apache.spark.sql.catalyst.util.IntervalUtils._
import org.apache.spark.sql.catalyst.util.TypeUtils.ordinalNumber
import org.apache.spark.sql.errors.DataTypeErrorsBase
@@ -1184,6 +1184,39 @@ class CollectionExpressionsSuite
Seq(-1.toByte, -2.toByte, -3.toByte))
}
+ test("SPARK-58440: illegal sequence boundaries carry String message
parameters") {
+ // Codegen-only: the interpreted path reports this through `require`,
which throws a plain
+ // IllegalArgumentException with no error class or parameters, so the
two-mode
+ // `checkErrorInExpression` cannot be used here.
+ // The parameter map is built in generated Java source. Janino erases the
+ // `Map<String, String>` type arguments, so non-String values used to slip
in and reach
+ // `SparkThrowable.getMessageParameters`, whose declared value type is
String.
+ withSQLConf(
+ SQLConf.CODEGEN_FACTORY_MODE.key ->
CodegenObjectFactoryMode.CODEGEN_ONLY.toString) {
+ // Numeric start/stop/step (IntegralSequenceImpl).
+ checkError(
+ exception = intercept[SparkIllegalArgumentException] {
+ evaluateWithMutableProjection(new Sequence(Literal(1), Literal(2),
Literal(0)))
+ },
+ condition = "_LEGACY_ERROR_TEMP_3243",
+ parameters = Map("start" -> "1", "stop" -> "2", "step" -> "0"))
+
+ // Interval step (InternalSequenceBase): `step` is a CalendarInterval,
not a number. A
+ // month-granularity step is required to reach this path; a
day-granularity one is
+ // delegated to the integral implementation with a plain `int` step.
+ checkError(
+ exception = intercept[SparkIllegalArgumentException] {
+ evaluateWithMutableProjection(Sequence(
+ Literal(Date.valueOf("1970-01-01")),
+ Literal(Date.valueOf("1970-02-01")),
+ Some(Literal(negateExact(stringToInterval("interval 1 month")))),
+ UTC_OPT))
+ },
+ condition = "_LEGACY_ERROR_TEMP_3243",
+ parameters = Map("start" -> "0", "stop" -> "2678400000000", "step" ->
"-1 months"))
+ }
+ }
+
test("Sequence of timestamps") {
checkEvaluation(new Sequence(
Literal(Timestamp.valueOf("2018-01-01 00:00:00")),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]