This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new f1402dbb11af [SPARK-57955][SQL] Raise a proper error for
out-of-Int-range data type parameters
f1402dbb11af is described below
commit f1402dbb11af775d6e4d89626449ab7b78ea6f7a
Author: YangJie <[email protected]>
AuthorDate: Tue Jul 28 22:52:05 2026 +0800
[SPARK-57955][SQL] Raise a proper error for out-of-Int-range data type
parameters
### What changes were proposed in this pull request?
The data type parser converts integer type parameters (DECIMAL
precision/scale, CHAR/VARCHAR length, TIME precision, GEOMETRY/GEOGRAPHY SRID)
to `Int` with a raw `.toInt`. The grammar backs these with `INTEGER_VALUE :
DIGIT+` (an unbounded digit run), so a value outside the 32-bit integer range
overflows and throws a raw `java.lang.NumberFormatException` with no Spark
error class.
This guards the conversion on all three data-type parse paths so an
out-of-range parameter raises a proper Spark error:
- `DataTypeAstBuilder` (the ANTLR parser, used by SQL/CAST and
`DataTypeParser.parseDataType`)
- `DataType.nameToType` (the JSON path, `DataType.fromJson`)
- `LegacyTypeStringParser` (the case-class string parser kept for Spark 1.1
and earlier Parquet compatibility)
Routing:
- DECIMAL precision reuses `DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION`
- TIME precision reuses `UNSUPPORTED_TIME_PRECISION`
- GEOMETRY/GEOGRAPHY SRID reuses `ST_INVALID_SRID_VALUE`
- CHAR/VARCHAR length and DECIMAL scale use a new
`DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE` error condition
The `TIMESTAMP(p)` branch of the same parser already guarded this (raising
`INVALID_TIMESTAMP_PRECISION`); the other type parameters are now brought in
line. The unsupported-type branch additionally renders the raw token text
instead of parsing it to `Int`, so an oversized parameter on an unsupported
type (e.g. `FOO(9999999999)`) no longer leaks a `NumberFormatException` while
building the `UNSUPPORTED_DATATYPE` message.
### Why are the changes needed?
An out-of-`Int`-range type parameter surfaces a raw
`java.lang.NumberFormatException` that is not a `SparkThrowable` -- it has no
error condition and no SQLSTATE, so programmatic error handling (JDBC/Connect
clients that read the condition/SQLSTATE) gets nothing, and callers that catch
the parser's expected error surface may mistake it for an internal failure. For
example:
```sql
SELECT CAST(1 AS DECIMAL(9999999999, 2));
```
throws:
```
java.lang.NumberFormatException: For input string: "9999999999"
```
The same happens via `StructType.fromDDL`, `DataType.fromJson`,
`DataTypeParser.parseDataType`, and the legacy Parquet schema-string parser.
This mirrors the fix in SPARK-56395 for the `NEAREST BY` num_results literal,
which surfaced the same raw-`NumberFormatException` anti-pattern.
### Does this PR introduce _any_ user-facing change?
Yes. An out-of-`Int`-range data type parameter now raises a Spark error
with a proper error condition instead of a raw `NumberFormatException`. For
example, `CAST(1 AS DECIMAL(9999999999, 2))` now raises
`DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION`, and `CAST(1 AS CHAR(9999999999))`
raises the new `DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE`. In-range values are
unaffected.
### How was this patch tested?
A new test in `DataTypeParserSuite` covering DECIMAL precision
(scale-present and scale-absent), DECIMAL scale, CHAR/VARCHAR length (bare and
`COLLATE`), TIME precision, GEOMETRY/GEOGRAPHY SRID, and an unsupported
parameterized type, across the ANTLR, JSON, and legacy parse paths, plus a
valid `Int.MaxValue` boundary case (`CHAR(2147483647)`). `catalyst/testOnly
*DataTypeParserSuite` and `core/testOnly org.apache.spark.SparkThrowableSuite`
pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code Opus 4.8
Closes #57034 from LuciferYang/SPARK-datatype-param-overflow.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit fd5b0d48f7551fe38156be2069d06f27812ecf0e)
Signed-off-by: yangjie01 <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 6 ++
.../sql/catalyst/parser/DataTypeAstBuilder.scala | 69 ++++++++++--
.../catalyst/parser/LegacyTypeStringParser.scala | 19 +++-
.../apache/spark/sql/errors/DataTypeErrors.scala | 39 ++++++-
.../org/apache/spark/sql/types/DataType.scala | 43 +++++++-
.../sql/catalyst/parser/DataTypeParserSuite.scala | 116 ++++++++++++++++++++-
6 files changed, 274 insertions(+), 18 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 54a65e965867..cb50b02cf5d7 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -2129,6 +2129,12 @@
],
"sqlState" : "42K01"
},
+ "DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE" : {
+ "message" : [
+ "The <parameter> value <value> of the <type> data type is out of the
range of a 32-bit integer."
+ ],
+ "sqlState" : "22003"
+ },
"DATA_SOURCE_ALREADY_EXISTS" : {
"message" : [
"Data source '<provider>' already exists. Please choose a different name
for the new data source."
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala
index 757e1d06bfd2..e3186fd43cf1 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala
@@ -311,12 +311,12 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
ctx)
case (Some(length), None) =>
- CharType(length.getText.toInt)
+ CharType(parseIntTypeParameter(length.getText, "length", "CHAR"))
case (Some(length), Some(collate)) =>
val collationId =
CollationFactory.fullyQualifiedNameToId(visitCollateClause(collate).toArray)
- CharType(length.getText.toInt, collationId)
+ CharType(parseIntTypeParameter(length.getText, "length",
"CHAR"), collationId)
}
case VARCHAR =>
(Option(currentCtx.length), Option(currentCtx.collateClause)) match {
@@ -326,20 +326,22 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
ctx)
case (Some(length), None) =>
- VarcharType(length.getText.toInt)
+ VarcharType(parseIntTypeParameter(length.getText, "length",
"VARCHAR"))
case (Some(length), Some(collate)) =>
val collationId =
CollationFactory.fullyQualifiedNameToId(visitCollateClause(collate).toArray)
- VarcharType(length.getText.toInt, collationId)
+ VarcharType(parseIntTypeParameter(length.getText, "length",
"VARCHAR"), collationId)
}
case DECIMAL | DEC | NUMERIC =>
if (currentCtx.precision == null) {
DecimalType.USER_DEFAULT
} else if (currentCtx.scale == null) {
- DecimalType(currentCtx.precision.getText.toInt, 0)
+ DecimalType(parseDecimalPrecision(currentCtx.precision.getText), 0)
} else {
- DecimalType(currentCtx.precision.getText.toInt,
currentCtx.scale.getText.toInt)
+ DecimalType(
+ parseDecimalPrecision(currentCtx.precision.getText),
+ parseIntTypeParameter(currentCtx.scale.getText, "scale",
"DECIMAL"))
}
case INTERVAL =>
if (currentCtx.fromDayTime != null) {
@@ -390,7 +392,7 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
val precision = if (currentCtx.precision == null) {
TimeType.DEFAULT_PRECISION
} else {
- currentCtx.precision.getText.toInt
+ parseTimePrecision(currentCtx.precision.getText)
}
TimeType(precision)
case GEOGRAPHY =>
@@ -404,7 +406,7 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
} else {
// The explicitly parameterzied GEOGRAPHY syntax uses a specified
integer SRID value.
// This implies a fixed GEOGRAPHY type, with a single fixed SRID
value across all rows.
- GeographyType(currentCtx.srid.getText.toInt)
+ GeographyType(parseSrid(currentCtx.srid.getText))
}
case GEOMETRY =>
// Unparameterized geometry type isn't supported and will be caught
by the default branch.
@@ -416,7 +418,7 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
} else {
// The explicitly parameterzied GEOMETRY type syntax has a single
integer SRID value.
// This implies a fixed GEOMETRY type, with a single fixed SRID
value across all rows.
- GeometryType(currentCtx.srid.getText.toInt)
+ GeometryType(parseSrid(currentCtx.srid.getText))
}
}
} else if (typeCtx.trivialPrimitiveType != null) {
@@ -436,10 +438,13 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
}
} else {
val badType = typeCtx.unsupportedType.getText
+ // Render the raw token text for the error message rather than routing
through
+ // `getIntegerValue` (which parses to `Int`): the type is unsupported
regardless, and a huge
+ // parameter such as `FOO(9999999999)` must not leak a raw
`NumberFormatException` here.
val params = typeCtx
.integerValue()
.asScala
- .map(getIntegerValue(_).toString)
+ .map(_.getText)
.toList
val dtStr =
if (params.nonEmpty) s"$badType(${params.mkString(",")})"
@@ -478,6 +483,50 @@ class DataTypeAstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with DataTypeE
}
}
+ // Parses an integer data-type parameter (CHAR/VARCHAR length or DECIMAL
scale). The grammar
+ // backs these with `INTEGER_VALUE : DIGIT+` (an unbounded digit run), so a
value outside the
+ // 32-bit integer range surfaces a proper Spark error instead of a raw
`NumberFormatException`.
+ private def parseIntTypeParameter(text: String, parameter: String, typeName:
String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.datatypeParameterValueOutOfRangeError(parameter,
text, typeName)
+ }
+ }
+
+ // Parses a DECIMAL precision, reusing
DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION: a value outside
+ // the 32-bit integer range is, a fortiori, beyond the maximum decimal
precision.
+ private def parseDecimalPrecision(text: String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.decimalPrecisionExceedsMaxPrecisionError(
+ text,
+ DecimalType.MAX_PRECISION)
+ }
+ }
+
+ // Parses a TIME precision, reusing UNSUPPORTED_TIME_PRECISION: a value
outside the 32-bit
+ // integer range is out of the supported precision range.
+ private def parseTimePrecision(text: String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.unsupportedTimePrecisionError(text)
+ }
+ }
+
+ // Parses a GEOMETRY/GEOGRAPHY SRID, reusing ST_INVALID_SRID_VALUE: a value
outside the 32-bit
+ // integer range is an unsupported SRID, matching the error an in-range but
unsupported SRID
+ // already produces.
+ private def parseSrid(text: String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.stInvalidSridValueError(text)
+ }
+ }
+
private def parseTimestampLtzNanosPrecision(precision: String): DataType = {
val p =
try precision.toInt
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/LegacyTypeStringParser.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/LegacyTypeStringParser.scala
index 7d1986e727f7..ffa9ef9e589c 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/LegacyTypeStringParser.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/LegacyTypeStringParser.scala
@@ -47,7 +47,24 @@ object LegacyTypeStringParser extends RegexParsers {
protected lazy val fixedDecimalType: Parser[DataType] =
("DecimalType(" ~> "[0-9]+".r) ~ ("," ~> "[0-9]+".r <~ ")") ^^ { case
precision ~ scale =>
- DecimalType(precision.toInt, scale.toInt)
+ // The captures are `[0-9]+` (an unbounded digit run), so a value
outside the 32-bit integer
+ // range would overflow `Int`. Guard the conversion to surface a proper
Spark error instead
+ // of a raw `NumberFormatException`, mirroring the parser and JSON paths.
+ val p =
+ try precision.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.decimalPrecisionExceedsMaxPrecisionError(
+ precision,
+ DecimalType.MAX_PRECISION)
+ }
+ val s =
+ try scale.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw
DataTypeErrors.datatypeParameterValueOutOfRangeError("scale", scale, "DECIMAL")
+ }
+ DecimalType(p, s)
}
protected lazy val arrayType: Parser[DataType] =
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
index 8d65217fb4e4..326d918f2486 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala
@@ -38,10 +38,15 @@ private[sql] object DataTypeErrors extends
DataTypeErrorsBase {
def decimalPrecisionExceedsMaxPrecisionError(
precision: Int,
maxPrecision: Int): SparkArithmeticException = {
+ decimalPrecisionExceedsMaxPrecisionError(precision.toString, maxPrecision)
+ }
+
+ def decimalPrecisionExceedsMaxPrecisionError(
+ precision: String,
+ maxPrecision: Int): SparkArithmeticException = {
new SparkArithmeticException(
errorClass = "DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION",
- messageParameters =
- Map("precision" -> precision.toString, "maxPrecision" ->
maxPrecision.toString),
+ messageParameters = Map("precision" -> precision, "maxPrecision" ->
maxPrecision.toString),
context = Array.empty,
summary = "")
}
@@ -286,12 +291,40 @@ private[sql] object DataTypeErrors extends
DataTypeErrorsBase {
}
def unsupportedTimePrecisionError(precision: Int): Throwable = {
+ unsupportedTimePrecisionError(precision.toString)
+ }
+
+ def unsupportedTimePrecisionError(precision: String): Throwable = {
new SparkException(
errorClass = "UNSUPPORTED_TIME_PRECISION",
- messageParameters = Map("precision" -> precision.toString),
+ messageParameters = Map("precision" -> precision),
cause = null)
}
+ // Raised when an integer type parameter (CHAR/VARCHAR length or DECIMAL
scale) is a well-formed
+ // digit sequence outside the range of a 32-bit integer. The grammar accepts
an unbounded run of
+ // digits, so this guards the parameter conversion and surfaces a proper
error instead of a raw
+ // `NumberFormatException`. DECIMAL precision and TIME precision reuse their
own dedicated errors;
+ // GEOMETRY/GEOGRAPHY SRID reuses `ST_INVALID_SRID_VALUE`.
+ def datatypeParameterValueOutOfRangeError(
+ parameter: String,
+ value: String,
+ typeName: String): Throwable = {
+ new SparkException(
+ errorClass = "DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE",
+ messageParameters = Map("parameter" -> parameter, "value" -> value,
"type" -> typeName),
+ cause = null)
+ }
+
+ // Raised when a GEOMETRY/GEOGRAPHY SRID literal is a well-formed digit
sequence outside the range
+ // of a 32-bit integer. Reuses `ST_INVALID_SRID_VALUE` so an overflowing
SRID surfaces the same
+ // error as an in-range but unsupported SRID, instead of a raw
`NumberFormatException`.
+ def stInvalidSridValueError(srid: String): Throwable = {
+ new SparkIllegalArgumentException(
+ errorClass = "ST_INVALID_SRID_VALUE",
+ messageParameters = Map("srid" -> srid))
+ }
+
def invalidTimestampPrecisionError(precision: String, typeName: String):
Throwable = {
new SparkException(
errorClass = "INVALID_TIMESTAMP_PRECISION",
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
index c8621ca4da17..dc79267e120c 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
@@ -215,13 +215,50 @@ object DataType {
.toMap
}
+ /**
+ * Parses an integer type parameter captured from a JSON type name (e.g. the
length in
+ * `char(10)`). The capture group is `\d+`, an unbounded digit run, so a
value outside the
+ * 32-bit integer range surfaces a proper Spark error instead of a raw
`NumberFormatException`.
+ * Mirrors the guard on the parser path in `DataTypeAstBuilder`.
+ */
+ private def parseIntTypeParameterFromJson(
+ text: String,
+ parameter: String,
+ typeName: String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.datatypeParameterValueOutOfRangeError(parameter,
text, typeName)
+ }
+ }
+
+ /**
+ * Parses the precision captured from a `decimal(p,s)` JSON type name,
reusing the dedicated
+ * `DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION` error when the value is outside
the 32-bit integer
+ * range.
+ */
+ private def parseDecimalPrecisionFromJson(text: String): Int = {
+ try text.toInt
+ catch {
+ case _: NumberFormatException =>
+ throw DataTypeErrors.decimalPrecisionExceedsMaxPrecisionError(
+ text,
+ DecimalType.MAX_PRECISION)
+ }
+ }
+
/** Given the string representation of a type, return its DataType */
private def nameToType(name: String): DataType = {
name match {
case "decimal" => DecimalType.USER_DEFAULT
- case FIXED_DECIMAL(precision, scale) => DecimalType(precision.toInt,
scale.toInt)
- case CHAR_TYPE(length) => CharType(length.toInt)
- case VARCHAR_TYPE(length) => VarcharType(length.toInt)
+ case FIXED_DECIMAL(precision, scale) =>
+ DecimalType(
+ parseDecimalPrecisionFromJson(precision),
+ parseIntTypeParameterFromJson(scale, "scale", "DECIMAL"))
+ case CHAR_TYPE(length) =>
+ CharType(parseIntTypeParameterFromJson(length, "length", "CHAR"))
+ case VARCHAR_TYPE(length) =>
+ VarcharType(parseIntTypeParameterFromJson(length, "length", "VARCHAR"))
case STRING_WITH_COLLATION(collation) => StringType(collation)
// If the coordinate reference system (CRS) value is omitted, Parquet
and other storage
// formats (Delta, Iceberg) consider "OGC:CRS84" to be the default value
of the crs.
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
index 6f9488ab4e52..724b11749534 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.catalyst.parser
-import org.apache.spark.{SparkException, SparkFunSuite}
+import org.apache.spark.{SparkArithmeticException, SparkException,
SparkFunSuite, SparkIllegalArgumentException}
import org.apache.spark.sql.catalyst.plans.SQLHelper
import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.internal.SQLConf
@@ -502,4 +502,118 @@ class DataTypeParserSuite extends SparkFunSuite with
SQLHelper {
}
}
}
+
+ test("SPARK-57955: integer type parameters that overflow Int surface a
proper error") {
+ // The grammar backs length/precision/scale/SRID with `INTEGER_VALUE :
DIGIT+` (an unbounded
+ // digit run), so a value > Int.MaxValue overflows Int. It must raise a
Spark error with an
+ // error class -- mirroring the existing TIMESTAMP(p) behavior -- rather
than a raw
+ // NumberFormatException. `tooLarge` is Int.MaxValue + 1, the first value
that overflows.
+ val tooLarge = (Int.MaxValue.toLong + 1).toString // 2147483648
+
+ // A parameter equal to Int.MaxValue does NOT overflow: it parses to an
Int and is then handled
+ // by each type's own validation, so no
DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE is raised for it.
+ // CHAR(Int.MaxValue) is a valid declaration (only length >= 0 is checked
at construction).
+ assert(CatalystSqlParser.parseDataType(s"CHAR(${Int.MaxValue})") ===
CharType(Int.MaxValue))
+
+ // DECIMAL precision reuses DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION (a
+ // SparkArithmeticException): an overflowing value trivially exceeds the
max precision of 38,
+ // matching the error DECIMAL(50) already produces. Covers both the
scale-present and
+ // scale-absent call sites.
+ Seq(s"DECIMAL($tooLarge, 2)", s"DECIMAL($tooLarge)").foreach { typeString
=>
+ checkError(
+ exception = intercept[SparkArithmeticException] {
+ CatalystSqlParser.parseDataType(typeString)
+ },
+ condition = "DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION",
+ parameters = Map("precision" -> tooLarge, "maxPrecision" -> "38"))
+ }
+
+ // DECIMAL scale and CHAR/VARCHAR length use the generic
DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE.
+ // Both the bare and the COLLATE-qualified CHAR/VARCHAR branches are
exercised.
+ Seq(
+ s"DECIMAL(10, $tooLarge)" -> ("scale", "DECIMAL"),
+ s"CHAR($tooLarge)" -> ("length", "CHAR"),
+ s"CHAR($tooLarge) COLLATE UTF8_BINARY" -> ("length", "CHAR"),
+ s"VARCHAR($tooLarge)" -> ("length", "VARCHAR"),
+ s"VARCHAR($tooLarge) COLLATE UTF8_BINARY" -> ("length",
"VARCHAR")).foreach {
+ case (typeString, (parameter, typeName)) =>
+ checkError(
+ exception = intercept[SparkException] {
+ CatalystSqlParser.parseDataType(typeString)
+ },
+ condition = "DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE",
+ parameters = Map(
+ "parameter" -> parameter,
+ "value" -> tooLarge,
+ "type" -> typeName))
+ }
+
+ // TIME precision reuses UNSUPPORTED_TIME_PRECISION.
+ checkError(
+ exception = intercept[SparkException] {
+ CatalystSqlParser.parseDataType(s"TIME($tooLarge)")
+ },
+ condition = "UNSUPPORTED_TIME_PRECISION",
+ parameters = Map("precision" -> tooLarge))
+
+ // GEOMETRY/GEOGRAPHY SRID reuses ST_INVALID_SRID_VALUE (a
SparkIllegalArgumentException): an
+ // overflowing SRID surfaces the same error as an in-range but unsupported
SRID.
+ Seq(s"GEOMETRY($tooLarge)", s"GEOGRAPHY($tooLarge)").foreach { typeString
=>
+ checkError(
+ exception = intercept[SparkIllegalArgumentException] {
+ CatalystSqlParser.parseDataType(typeString)
+ },
+ condition = "ST_INVALID_SRID_VALUE",
+ parameters = Map("srid" -> tooLarge))
+ }
+
+ // An unsupported parameterized type must not leak a raw
NumberFormatException from rendering
+ // its (oversized) parameter into the error message.
+ checkError(
+ exception = intercept[ParseException] {
+ CatalystSqlParser.parseDataType(s"FOO($tooLarge)")
+ },
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> s""""FOO($tooLarge)""""))
+
+ // The JSON path (DataType.fromJson) is guarded consistently with the
parser path, for every
+ // capture that converts to Int: decimal precision, decimal scale, char
length, varchar length.
+ checkError(
+ exception = intercept[SparkArithmeticException] {
+ DataType.fromJson(s""""decimal($tooLarge,2)"""")
+ },
+ condition = "DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION",
+ parameters = Map("precision" -> tooLarge, "maxPrecision" -> "38"))
+ Seq(
+ s"decimal(10,$tooLarge)" -> ("scale", "DECIMAL"),
+ s"char($tooLarge)" -> ("length", "CHAR"),
+ s"varchar($tooLarge)" -> ("length", "VARCHAR")).foreach {
+ case (jsonName, (parameter, typeName)) =>
+ checkError(
+ exception = intercept[SparkException] {
+ DataType.fromJson(s""""$jsonName"""")
+ },
+ condition = "DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE",
+ parameters = Map(
+ "parameter" -> parameter,
+ "value" -> tooLarge,
+ "type" -> typeName))
+ }
+
+ // The legacy case-class type-string parser (Spark <= 1.1 Parquet
compatibility) is guarded the
+ // same way. It is only reachable as a fallback from StructType.fromString
when JSON parsing
+ // fails, so it must not leak a raw NumberFormatException either.
+ checkError(
+ exception = intercept[SparkArithmeticException] {
+ LegacyTypeStringParser.parseString(s"DecimalType($tooLarge,2)")
+ },
+ condition = "DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION",
+ parameters = Map("precision" -> tooLarge, "maxPrecision" -> "38"))
+ checkError(
+ exception = intercept[SparkException] {
+ LegacyTypeStringParser.parseString(s"DecimalType(10,$tooLarge)")
+ },
+ condition = "DATATYPE_PARAMETER_VALUE_OUT_OF_RANGE",
+ parameters = Map("parameter" -> "scale", "value" -> tooLarge, "type" ->
"DECIMAL"))
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]