Prashanth Babu created SPARK-34679:
--------------------------------------
Summary: inferTimestamp option is missing from the list of options
in DataFrameReader.json.
Key: SPARK-34679
URL: https://issues.apache.org/jira/browse/SPARK-34679
Project: Spark
Issue Type: Documentation
Components: Documentation
Affects Versions: 3.1.1, 3.1.0, 3.0.2, 3.0.1
Reporter: Prashanth Babu
{color:#e01e5a}inferTimestamp{color} option is missing in the list of options
in {color:#e01e5a}DataFrameReader.json{color} method in the API docs missing
from the [Scaladocs
here|[DataFrameReader.json|https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L432-L520]].
Simiarly in the [Pyspark
docs|[pyspark.sql.DataFrameReader.json|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrameReader.json.html?highlight=json#pyspark.sql.DataFrameReader.json]]
as well.
However we have this blurb in the [migration guide|[Spark 3.0 to 3.0.1
migration
guide|https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-30-to-301]]
* In Spark 3.0, JSON datasource and JSON function {{schema_of_json}} infer
TimestampType from string values if they match to the pattern defined by the
JSON option {{timestampFormat}}. Since version 3.0.1, the timestamp type
inference is disabled by default. Set the JSON option {{inferTimestamp}} to
{{true}} to enable such type inference.
We should add this in the documentation as well as there is a possibility that
the Data Engineers might not be aware of this option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
