Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21217#discussion_r186276456
--- Diff: docs/sql-programming-guide.md ---
@@ -1812,6 +1812,7 @@ working with timestamps in `pandas_udf`s to get the
best performance, see
- Since Spark 2.4, creating a managed table with nonempty location is
not allowed. An exception is thrown when attempting to create a managed table
with nonempty location. To set `true` to
`spark.sql.allowCreatingManagedTableUsingNonemptyLocation` restores the
previous behavior. This option will be removed in Spark 3.0.
- Since Spark 2.4, the type coercion rules can automatically promote the
argument types of the variadic SQL functions (e.g., IN/COALESCE) to the widest
common type, no matter how the input arguments order. In prior Spark versions,
the promotion could fail in some specific orders (e.g., TimestampType,
IntegerType and StringType) and throw an exception.
- In version 2.3 and earlier, `to_utc_timestamp` and
`from_utc_timestamp` respect the timezone in the input timestamp string, which
breaks the assumption that the input timestamp is in a specific timezone.
Therefore, these 2 functions can return unexpected results. In version 2.4 and
later, this problem has been fixed. `to_utc_timestamp` and `from_utc_timestamp`
will return null if the input timestamp string contains timezone. As an
example, `from_utc_timestamp('2000-10-10 00:00:00', 'GMT+1')` will return
`2000-10-10 01:00:00` in both Spark 2.3 and 2.4. However,
`from_utc_timestamp('2000-10-10 00:00:00+00:00', 'GMT+1')`, assuming a local
timezone of GMT+8, will return `2000-10-10 09:00:00` in Spark 2.3 but `null` in
2.4. For people who don't care about this problem and want to retain the
previous behaivor to keep their query unchanged, you can set
`spark.sql.function.rejectTimezoneInString` to false. This option will be
removed in Spark 3.0 and should only be used as a tempora
ry workaround.
+ - In version 2.3, if `spark.sql.caseSensitive` is set to true, then the
`CURRENT_DATE` and `CURRENT_TIMESTAMP` functions incorrectly became
case-sensitive and would resolve to columns (unless typed in lower case). In
Spark 2.4 this has been fixed and the functions are no longer case-sensitive.
--- End diff --
+1 for @felixcheung 's suggestion.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
