This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new a53394f  [SPARK-31874][SQL] Use `FastDateFormat` as the legacy 
fractional formatter
a53394f is described below

commit a53394fda600279923c7ac7e61c1d763993e680b
Author: Max Gekk <max.g...@gmail.com>
AuthorDate: Sun May 31 13:05:00 2020 +0000

    [SPARK-31874][SQL] Use `FastDateFormat` as the legacy fractional formatter
    
    ### What changes were proposed in this pull request?
    1. Replace `SimpleDateFormat` by `FastDateFormat` as the legacy formatter 
of `FractionTimestampFormatter`.
    2. Optimise `LegacyFastTimestampFormatter` for `java.sql.Timestamp` w/o 
fractional part.
    
    ### Why are the changes needed?
    1. By default `HiveResult`.`hiveResultString` retrieves timestamps values 
as instances of `java.sql.Timestamp`, and uses the legacy parser 
`SimpleDateFormat` to convert the timestamps to strings. After the fix 
https://github.com/apache/spark/pull/28024, the fractional formatter and its 
companion - legacy formatter `SimpleDateFormat` are created per every value. By 
switching from `LegacySimpleTimestampFormatter` to 
`LegacyFastTimestampFormatter`, we can utilize the internal cache of `F [...]
    2. The second change in the method `def format(ts: Timestamp): String` of 
`LegacyFastTimestampFormatter` is needed to optimize the formatter for patterns 
without the fractional part and avoid conversions to microseconds.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    By existing tests in `TimestampFormatter`.
    
    Closes #28678 from MaxGekk/fastdateformat-as-legacy-frac-formatter.
    
    Authored-by: Max Gekk <max.g...@gmail.com>
    Signed-off-by: Wenchen Fan <wenc...@databricks.com>
    (cherry picked from commit 47dc332258bec20c460f666de50d9a8c5c0fbc0a)
    Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
 .../org/apache/spark/sql/catalyst/util/TimestampFormatter.scala    | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
index 32b4dcd..6d1c535 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
@@ -121,6 +121,7 @@ class FractionTimestampFormatter(zoneId: ZoneId)
     TimestampFormatter.defaultPattern,
     zoneId,
     TimestampFormatter.defaultLocale,
+    LegacyDateFormats.FAST_DATE_FORMAT,
     needVarLengthSecondFraction = false) {
 
   @transient
@@ -203,7 +204,11 @@ class LegacyFastTimestampFormatter(
   }
 
   override def format(ts: Timestamp): String = {
-    format(fromJavaTimestamp(ts))
+    if (ts.getNanos == 0) {
+      fastDateFormat.format(ts)
+    } else {
+      format(fromJavaTimestamp(ts))
+    }
   }
 
   override def format(instant: Instant): String = {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to