Yaoxuan Wu created SPARK-60333:
----------------------------------
Summary: make_timestamp wraps out-of-range seconds into a valid
time (interpreted path)
Key: SPARK-60333
URL: https://issues.apache.org/jira/browse/SPARK-60333
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
make_timestamp returns a wrong timestamp instead of NULL or an error when the
seconds overflow an int (interpreted evaluation and constant folding)
{code:java}
SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, 4324.967296);
-- Spark 4.2.0: 2020-01-01 00:00:30
-- expected: NULL (spark.sql.ansi.enabled=false), or
DATETIME_FIELD_OUT_OF_BOUNDS (ANSI)
SELECT make_timestamp_ntz(2020, 1, 1, 0, 0, 61);
-- NULL / DATETIME_FIELD_OUT_OF_BOUNDS (correct) {code}
With ANSI on, the first query also returns 2020-01-01 00:00:30 instead of
raising an error.The result depends on the evaluation path. On a column,
codegen is correct and interpreted evaluation is not:
{code:java}
from decimal import Decimal
spark.conf.set("spark.sql.ansi.enabled", "false")
spark.createDataFrame([(Decimal("4324.967296"),), (Decimal("-2147483648"),),
(Decimal("61"),)],
"s decimal(16,6)").createOrReplaceTempView("t")
q = "SELECT s, make_timestamp_ntz(2020, 1, 1, 0, 0, s) FROM t ORDER BY s"
spark.sql(q).collect()
# codegen (default): NULL, NULL, NULL
spark.conf.set("spark.sql.codegen.wholeStage", "false")
spark.conf.set("spark.sql.codegen.factoryMode", "NO_CODEGEN")
spark.sql(q).collect()
# interpreted: -2147483648 -> 2020-01-01 00:00:00, 61 -> NULL, 4324.967296 ->
2020-01-01 00:00:30 {code}
The same holds for {{{}make_timestamp{}}}, {{make_timestamp_ltz}} and the
{{try_}} variants. Constant folding evaluates the interpreted path, which is
why the literal query above is wrong with default settings.
*Cause:* {{sec}} is DECIMAL(16, 6), but the interpreted
{{MakeTimestamp.toMicros}} narrows the microseconds with
{{secAndMicros.toUnscaledLong.toInt}} ("8 digits cannot overflow Int", from
SPARK-32043). Any |sec| >= 2147.483648 wraps modulo 2^32 microseconds into a
second in [0, 60), so the range check never sees it: 4324.967296 s = 2^32 +
30,000,000 µs becomes 30 s, and -2147483648 s (a multiple of 2^32 µs) becomes 0
s. The codegen path uses {{sec.floor().toInt()}} and returns NULL. On master
the same {{(int)}} narrowing is in
{{{}DateTimeExpressionUtils.makeTimestampMicros{}}}, which codegen now calls
too, so both paths may return the wrapped value there
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
