HyukjinKwon commented on a change in pull request #31489:
URL: https://github.com/apache/spark/pull/31489#discussion_r571539866
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -825,6 +825,26 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
* must be in the following form: YYYY-MM-DDTHH:mm:ss (e.g.
2020-06-01T13:00:00)</li>
* <li>`recursiveFileLookup`: recursively scan a directory for files. Using
this option
* disables partition discovery</li>
+ * <li>`datetimeRebaseMode`: the rebasing mode for the values of the `DATE`,
`TIMESTAMP_MICROS`,
+ * `TIMESTAMP_MILLIS` logical types from the Julian to Proleptic Gregorian
calendar:
+ * <ul>
+ * <li>`EXCEPTION` : Spark fails in reads of ancient dates/timestamps
that are ambiguous
+ * between the two calendars</li>
+ * <li>`CORRECTED` : loading of dates/timestamps without rebasing</li>
+ * <li>`LEGACY` : perform rebasing of ancient dates/timestamps from the
Julian to Proleptic
+ * Gregorian calendar</li>
+ * </ul>
+ * </li>
+ * <li>`int96RebaseMode`: the rebasing mode for `INT96` timestamps from the
Julian to Proleptic
+ * Gregorian calendar:
Review comment:
Shall we mention that the default is the one set in
`spark.sql.legacy.parquet.datetimeRebaseModeInRead`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
