This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4ff2b94 [SPARK-26503][CORE][DOC][FOLLOWUP] Get rid of
spark.sql.legacy.timeParser.enabled
4ff2b94 is described below
commit 4ff2b94a7c827f9cc3e6c79fe090568d2743c0ca
Author: Maxim Gekk <max.g...@gmail.com>
AuthorDate: Sun Jan 13 11:20:22 2019 +0800
[SPARK-26503][CORE][DOC][FOLLOWUP] Get rid of
spark.sql.legacy.timeParser.enabled
## What changes were proposed in this pull request?
The SQL config `spark.sql.legacy.timeParser.enabled` was removed by
https://github.com/apache/spark/pull/23495. The PR cleans up the SQL migration
guide and the comment for `UnixTimestamp`.
Closes #23529 from MaxGekk/get-rid-off-legacy-parser-followup.
Authored-by: Maxim Gekk <max.g...@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/sql-migration-guide-upgrade.md | 4 ++--
.../apache/spark/sql/catalyst/expressions/datetimeExpressions.scala | 4 +---
2 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/docs/sql-migration-guide-upgrade.md
b/docs/sql-migration-guide-upgrade.md
index a2d782e..fce0b9a 100644
--- a/docs/sql-migration-guide-upgrade.md
+++ b/docs/sql-migration-guide-upgrade.md
@@ -33,13 +33,13 @@ displayTitle: Spark SQL Upgrading Guide
- In Spark version 2.4 and earlier, the `SET` command works without any
warnings even if the specified key is for `SparkConf` entries and it has no
effect because the command does not update `SparkConf`, but the behavior might
confuse users. Since 3.0, the command fails if a `SparkConf` key is used. You
can disable such a check by setting
`spark.sql.legacy.setCommandRejectsSparkCoreConfs` to `false`.
- - Since Spark 3.0, CSV/JSON datasources use java.time API for parsing and
generating CSV/JSON content. In Spark version 2.4 and earlier,
java.text.SimpleDateFormat is used for the same purpose with fallbacks to the
parsing mechanisms of Spark 2.0 and 1.x. For example, `2018-12-08 10:39:21.123`
with the pattern `yyyy-MM-dd'T'HH:mm:ss.SSS` cannot be parsed since Spark 3.0
because the timestamp does not match to the pattern but it can be parsed by
earlier Spark versions due to a fallback [...]
+ - Since Spark 3.0, CSV/JSON datasources use java.time API for parsing and
generating CSV/JSON content. In Spark version 2.4 and earlier,
java.text.SimpleDateFormat is used for the same purpose with fallbacks to the
parsing mechanisms of Spark 2.0 and 1.x. For example, `2018-12-08 10:39:21.123`
with the pattern `yyyy-MM-dd'T'HH:mm:ss.SSS` cannot be parsed since Spark 3.0
because the timestamp does not match to the pattern but it can be parsed by
earlier Spark versions due to a fallback [...]
- In Spark version 2.4 and earlier, CSV datasource converts a malformed CSV
string to a row with all `null`s in the PERMISSIVE mode. Since Spark 3.0, the
returned row can contain non-`null` fields if some of CSV column values were
parsed and converted to desired types successfully.
- In Spark version 2.4 and earlier, JSON datasource and JSON functions like
`from_json` convert a bad JSON record to a row with all `null`s in the
PERMISSIVE mode when specified schema is `StructType`. Since Spark 3.0, the
returned row can contain non-`null` fields if some of JSON column values were
parsed and converted to desired types successfully.
- - Since Spark 3.0, the `unix_timestamp`, `date_format`, `to_unix_timestamp`,
`from_unixtime`, `to_date`, `to_timestamp` functions use java.time API for
parsing and formatting dates/timestamps from/to strings by using ISO chronology
(https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html)
based on Proleptic Gregorian calendar. In Spark version 2.4 and earlier,
java.text.SimpleDateFormat and java.util.GregorianCalendar (hybrid calendar
that supports both the Julian [...]
+ - Since Spark 3.0, the `unix_timestamp`, `date_format`, `to_unix_timestamp`,
`from_unixtime`, `to_date`, `to_timestamp` functions use java.time API for
parsing and formatting dates/timestamps from/to strings by using ISO chronology
(https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html)
based on Proleptic Gregorian calendar. In Spark version 2.4 and earlier,
java.text.SimpleDateFormat and java.util.GregorianCalendar (hybrid calendar
that supports both the Julian [...]
- Since Spark 3.0, JSON datasource and JSON function `schema_of_json` infer
TimestampType from string values if they match to the pattern defined by the
JSON option `timestampFormat`. Set JSON option `inferTimestamp` to `false` to
disable such type inferring.
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 8fc0112..b1832da 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -614,9 +614,7 @@ case class ToUnixTimestamp(
/**
* Converts time string with given pattern to Unix time stamp (in seconds),
returns null if fail.
- * See
[http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]
- * if SQL config spark.sql.legacy.timeParser.enabled is set to true otherwise
- *
[https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html].
+ * See
[https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html].
* Note that hive Language Manual says it returns 0 if fail, but in fact it
returns null.
* If the second parameter is missing, use "yyyy-MM-dd HH:mm:ss".
* If no parameters provided, the first parameter will be current_timestamp.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
