[
https://issues.apache.org/jira/browse/SPARK-60484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130620#comment-18130620
]
Yahya Kisana commented on SPARK-60484:
--------------------------------------
I can take this
> CAST(string AS INTERVAL SECOND) drops the minus sign of values between -1 and
> 0 seconds
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-60484
> URL: https://issues.apache.org/jira/browse/SPARK-60484
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness
>
> {code:sql}
> SELECT CAST('-0.5' AS INTERVAL SECOND);
> -- expected: INTERVAL '-00.5' SECOND
> -- actual: INTERVAL '00.5' SECOND
> SELECT CAST('-1.5' AS INTERVAL SECOND);
> -- INTERVAL '-01.5' SECOND (correct)
> {code}
> Any value between -1 and 0 seconds loses its sign. Since the JSON and CSV
> readers parse INTERVAL SECOND with the same code, writing such a value to
> JSON or CSV and reading it back also turns -0.5 s into +0.5 s:
> {code:python}
> df = spark.sql("SELECT INTERVAL '-0.5' SECOND AS x")
> df.write.mode("overwrite").json("/tmp/iv")
> spark.read.schema(df.schema).json("/tmp/iv").first().x
> # datetime.timedelta(microseconds=500000), expected -500000
> {code}
> Cause: in {{IntervalUtils.parseSecondNano}}, the sign of the fraction is
> taken from the parsed whole seconds ({{parseNanos(nanosStr, seconds < 0)}}).
> For "-00.5" the whole seconds are 0, so the fraction is added as positive.
> The interval literal {{INTERVAL '-0.5' SECOND}} goes through the parser and
> is correct.
> Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
