Yaoxuan Wu created SPARK-60484:
----------------------------------
Summary: CAST(string AS INTERVAL SECOND) drops the minus sign of
values between -1 and 0 seconds
Key: SPARK-60484
URL: https://issues.apache.org/jira/browse/SPARK-60484
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
{code:sql}
SELECT CAST('-0.5' AS INTERVAL SECOND);
-- expected: INTERVAL '-00.5' SECOND
-- actual: INTERVAL '00.5' SECOND
SELECT CAST('-1.5' AS INTERVAL SECOND);
-- INTERVAL '-01.5' SECOND (correct)
{code}
Any value between -1 and 0 seconds loses its sign. Since the JSON and CSV
readers parse INTERVAL SECOND with the same code, writing such a value to JSON
or CSV and reading it back also turns -0.5 s into +0.5 s:
{code:python}
df = spark.sql("SELECT INTERVAL '-0.5' SECOND AS x")
df.write.mode("overwrite").json("/tmp/iv")
spark.read.schema(df.schema).json("/tmp/iv").first().x
# datetime.timedelta(microseconds=500000), expected -500000
{code}
Cause: in {{IntervalUtils.parseSecondNano}}, the sign of the fraction is taken
from the parsed whole seconds ({{parseNanos(nanosStr, seconds < 0)}}). For
"-00.5" the whole seconds are 0, so the fraction is added as positive. The
interval literal {{INTERVAL '-0.5' SECOND}} goes through the parser and is
correct.
Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
