[
https://issues.apache.org/jira/browse/SPARK-31758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-31758.
----------------------------------
Resolution: Cannot Reproduce
> Incorrect timestamp parsing from JSON
> -------------------------------------
>
> Key: SPARK-31758
> URL: https://issues.apache.org/jira/browse/SPARK-31758
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.5
> Reporter: Tomi Ruokola
> Priority: Major
>
> Parsing a json string into a TimestampType can give incorrect results.
> {code:python}
> schema = StructType([StructField("timestamp", TimestampType())])
> df = spark.createDataFrame([('{"timestamp":"2020-01-01T20:00:00.900125Z"}',
> )], ["body"])
> df.select(from_json("body", schema)).collect(){code}
> Output:
> {code:python}
> datetime.datetime(2020, 1, 1, 20, 15, 0, 125000){code}
> This seems to happen when the timestamp has sub-millisecond precision and a Z
> suffix. For example, if the fraction is .900125, the output fraction is .125
> and 900 seconds is added to the timestamp.
> Workaround: Adding the timestampFormat option fixes the problem, even if the
> format string is not exactly correct.
> {code:python}
> df.select(from_json("body", schema, {"timestampFormat": "yyyy-MM-dd
> HH:mm:ss"})).collect()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
