Veenit Shah created SPARK-26325:
-----------------------------------
Summary: Interpret timestamp fields in Spark while reading json
(timestampFormat)
Key: SPARK-26325
URL: https://issues.apache.org/jira/browse/SPARK-26325
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Veenit Shah
I am trying to read a pretty printed json which has time fields in it. I want
to interpret the timestamps columns as timestamp fields while reading the json
itself. However, it's still reading them as string when I {{printSchema}}
E.g. Input json file -
{code:java}
[{
"time_field" : "2017-09-30 04:53:39.412496Z"
}]
{code}
Code -
{code:java}
df = spark.read.option("multiLine",
"true").option("timestampFormat","yyyy-MM-dd
HH:mm:ss.SSSSSS'Z'").json('path_to_json_file')
{code}
Output of df.printSchema() -
{code:java}
root
|-- time_field: string (nullable = true)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
