[
https://issues.apache.org/jira/browse/SPARK-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801802#comment-16801802
]
Jurriaan Pruis commented on SPARK-27224:
----------------------------------------
[~hyukjin.kwon] is this the same as
https://issues.apache.org/jira/browse/SPARK-17914?focusedCommentId=16750292&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16750292
? We're also having this issue when upgrading to 2.4.0.
> Spark to_json parses UTC timestamp incorrectly
> ----------------------------------------------
>
> Key: SPARK-27224
> URL: https://issues.apache.org/jira/browse/SPARK-27224
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Jeff Xu
> Priority: Major
>
> When parsing ISO-8601 timestamp, if there is UTC suffix symbol, and more than
> 3 digits in the fraction part, from_json will give incorrect result.
>
> {noformat}
> scala> val schema = new StructType().add("t", TimestampType)
> #
> # no "Z", no problem
> #
> scala> val t = "2019-03-20T09:01:03.1234567"
> scala> Seq((s"""{"t":"${t}"}""")).toDF("json").select(from_json(col("json"),
> schema)).show(false)
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 09:01:03.123]|
> +-------------------------+
> #
> # Add "Z", incorrect
> #
> scala> val t = "2019-03-20T09:01:03.1234567Z"
> scala> Seq((s"""{"t":"${t}"}""")).toDF("json").select(from_json(col("json"),
> schema)).show(false)
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:21:37.567]|
> +-------------------------+
> #
> # reduce the # of digits, the conversion is incorrect until only we reach 3
> digits
> #
> scala> val t = "2019-03-20T09:01:03.123456Z"
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:03:06.456]|
> +-------------------------+
> scala> val t = "2019-03-20T09:01:03.12345Z
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:01:15.345]|
> +-------------------------+
> scala> val t = "2019-03-20T09:01:03.1234Z"
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:01:04.234]|
> +-------------------------+
> # correct when there is <=3 digits in fraction
> scala> val t = "2019-03-20T09:01:03.123Z"
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:01:03.123]|
> +-------------------------+
> scala> val t = "2019-03-20T09:01:03.999Z"
> +-------------------------+
> |jsontostructs(json) |
> +-------------------------+
> |[2019-03-20 02:01:03.999]|
> +-------------------------+
> {noformat}
>
> This could be related to SPARK-17914.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
