This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new a53394f [SPARK-31874][SQL] Use `FastDateFormat` as the legacy fractional formatter a53394f is described below commit a53394fda600279923c7ac7e61c1d763993e680b Author: Max Gekk <max.g...@gmail.com> AuthorDate: Sun May 31 13:05:00 2020 +0000 [SPARK-31874][SQL] Use `FastDateFormat` as the legacy fractional formatter ### What changes were proposed in this pull request? 1. Replace `SimpleDateFormat` by `FastDateFormat` as the legacy formatter of `FractionTimestampFormatter`. 2. Optimise `LegacyFastTimestampFormatter` for `java.sql.Timestamp` w/o fractional part. ### Why are the changes needed? 1. By default `HiveResult`.`hiveResultString` retrieves timestamps values as instances of `java.sql.Timestamp`, and uses the legacy parser `SimpleDateFormat` to convert the timestamps to strings. After the fix https://github.com/apache/spark/pull/28024, the fractional formatter and its companion - legacy formatter `SimpleDateFormat` are created per every value. By switching from `LegacySimpleTimestampFormatter` to `LegacyFastTimestampFormatter`, we can utilize the internal cache of `F [...] 2. The second change in the method `def format(ts: Timestamp): String` of `LegacyFastTimestampFormatter` is needed to optimize the formatter for patterns without the fractional part and avoid conversions to microseconds. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By existing tests in `TimestampFormatter`. Closes #28678 from MaxGekk/fastdateformat-as-legacy-frac-formatter. Authored-by: Max Gekk <max.g...@gmail.com> Signed-off-by: Wenchen Fan <wenc...@databricks.com> (cherry picked from commit 47dc332258bec20c460f666de50d9a8c5c0fbc0a) Signed-off-by: Wenchen Fan <wenc...@databricks.com> --- .../org/apache/spark/sql/catalyst/util/TimestampFormatter.scala | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala index 32b4dcd..6d1c535 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala @@ -121,6 +121,7 @@ class FractionTimestampFormatter(zoneId: ZoneId) TimestampFormatter.defaultPattern, zoneId, TimestampFormatter.defaultLocale, + LegacyDateFormats.FAST_DATE_FORMAT, needVarLengthSecondFraction = false) { @transient @@ -203,7 +204,11 @@ class LegacyFastTimestampFormatter( } override def format(ts: Timestamp): String = { - format(fromJavaTimestamp(ts)) + if (ts.getNanos == 0) { + fastDateFormat.format(ts) + } else { + format(fromJavaTimestamp(ts)) + } } override def format(instant: Instant): String = { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org