[
https://issues.apache.org/jira/browse/SPARK-60333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60333:
-----------------------------------
Labels: pull-request-available (was: )
> make_timestamp wraps out-of-range seconds into a valid time (interpreted path)
> ------------------------------------------------------------------------------
>
> Key: SPARK-60333
> URL: https://issues.apache.org/jira/browse/SPARK-60333
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: pull-request-available
>
> make_timestamp returns a wrong timestamp instead of NULL or an error when the
> seconds overflow an int (interpreted evaluation and constant folding)
>
> {code:java}
> SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, 4324.967296);
> -- Spark 4.2.0: 2020-01-01 00:00:30
> -- expected: NULL (spark.sql.ansi.enabled=false), or
> DATETIME_FIELD_OUT_OF_BOUNDS (ANSI)
> SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, 61);
> -- NULL / DATETIME_FIELD_OUT_OF_BOUNDS (correct) {code}
> With ANSI on, the first query also returns 2020-01-01 00:00:30 instead of
> raising an error.The result depends on the evaluation path. On a column,
> codegen is correct and interpreted evaluation is not:
> {code:java}
> from decimal import Decimal
> spark.conf.set("spark.sql.ansi.enabled", "false")
> spark.createDataFrame([(Decimal("4324.967296"),), (Decimal("-2147483648"),),
> (Decimal("61"),)],
> "s decimal(16,6)").createOrReplaceTempView("t")
> q = "SELECT s, make_timestamp_ntz(2020, 1, 1, 0, 0, s) FROM t ORDER BY s"
> spark.sql(q).collect()
> # codegen (default): NULL, NULL, NULL
> spark.conf.set("spark.sql.codegen.wholeStage", "false")
> spark.conf.set("spark.sql.codegen.factoryMode", "NO_CODEGEN")
> spark.sql(q).collect()
> # interpreted: -2147483648 -> 2020-01-01 00:00:00, 61 -> NULL, 4324.967296 ->
> 2020-01-01 00:00:30 {code}
> The same holds for {{{}make_timestamp{}}}, {{make_timestamp_ltz}} and the
> {{try_}} variants. Constant folding evaluates the interpreted path, which is
> why the literal query above is wrong with default settings.
> *Cause:* {{sec}} is DECIMAL(16, 6), but the interpreted
> {{MakeTimestamp.toMicros}} narrows the microseconds with
> {{secAndMicros.toUnscaledLong.toInt}} ("8 digits cannot overflow Int", from
> SPARK-32043). Any |sec| >= 2147.483648 wraps modulo 2^32 microseconds into a
> second in [0, 60), so the range check never sees it: 4324.967296 s = 2^32 +
> 30,000,000 µs becomes 30 s, and -2147483648 s (a multiple of 2^32 µs) becomes
> 0 s. The codegen path uses {{sec.floor().toInt()}} and returns NULL. On
> master the same {{(int)}} narrowing is in
> {{{}DateTimeExpressionUtils.makeTimestampMicros{}}}, which codegen now calls
> too, so both paths may return the wrapped value there
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
