[ https://issues.apache.org/jira/browse/SPARK-31758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-31758. ---------------------------------- Resolution: Cannot Reproduce > Incorrect timestamp parsing from JSON > ------------------------------------- > > Key: SPARK-31758 > URL: https://issues.apache.org/jira/browse/SPARK-31758 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.5 > Reporter: Tomi Ruokola > Priority: Major > > Parsing a json string into a TimestampType can give incorrect results. > {code:python} > schema = StructType([StructField("timestamp", TimestampType())]) > df = spark.createDataFrame([('{"timestamp":"2020-01-01T20:00:00.900125Z"}', > )], ["body"]) > df.select(from_json("body", schema)).collect(){code} > Output: > {code:python} > datetime.datetime(2020, 1, 1, 20, 15, 0, 125000){code} > This seems to happen when the timestamp has sub-millisecond precision and a Z > suffix. For example, if the fraction is .900125, the output fraction is .125 > and 900 seconds is added to the timestamp. > Workaround: Adding the timestampFormat option fixes the problem, even if the > format string is not exactly correct. > {code:python} > df.select(from_json("body", schema, {"timestampFormat": "yyyy-MM-dd > HH:mm:ss"})).collect() > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org