Jeff Xu created SPARK-27224:
-------------------------------
Summary: Spark to_json parses UTC timestamp incorrectly
Key: SPARK-27224
URL: https://issues.apache.org/jira/browse/SPARK-27224
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Jeff Xu
When parsing ISO-8601 timestamp, if there is UTC suffix symbol, and more than 3
digits in the fraction part, from_json will give incorrect result.
{noformat}
scala> val schema = new StructType().add("t", TimestampType)
#
# no "Z", no problem
#
scala> val t = "2019-03-20T09:01:03.1234567"
scala> Seq((s"""{"t":"${t}"}""")).toDF("json").select(from_json(col("json"),
schema)).show(false)
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 09:01:03.123]|
+-------------------------+
#
# Add "Z", incorrect
#
scala> val t = "2019-03-20T09:01:03.1234567Z"
scala> Seq((s"""{"t":"${t}"}""")).toDF("json").select(from_json(col("json"),
schema)).show(false)
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:21:37.567]|
+-------------------------+
#
# reduce the # of digits, the conversion is incorrect until only we reach 3
digits
#
scala> val t = "2019-03-20T09:01:03.123456Z"
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:03:06.456]|
+-------------------------+
scala> val t = "2019-03-20T09:01:03.12345Z
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:01:15.345]|
+-------------------------+
scala> val t = "2019-03-20T09:01:03.1234Z"
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:01:04.234]|
+-------------------------+
# correct when there is <=3 digits in fraction
scala> val t = "2019-03-20T09:01:03.123Z"
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:01:03.123]|
+-------------------------+
scala> val t = "2019-03-20T09:01:03.999Z"
+-------------------------+
|jsontostructs(json) |
+-------------------------+
|[2019-03-20 02:01:03.999]|
+-------------------------+
{noformat}
This could be related to SPARK-17914.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
