dongjoon-hyun commented on a change in pull request #28189: [SPARK-31426][SQL]
Fix perf regressions of toJavaTimestamp/fromJavaTimestamp
URL: https://github.com/apache/spark/pull/28189#discussion_r407874752
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -172,17 +176,8 @@ object DateTimeUtils {
* @return The number of micros since epoch from `java.sql.Timestamp`.
*/
def fromJavaTimestamp(t: Timestamp): SQLTimestamp = {
- val era = if (t.before(julianCommonEraStart)) 0 else 1
- val localDateTime = LocalDateTime.of(
- t.getYear + 1900, t.getMonth + 1, 1,
- t.getHours, t.getMinutes, t.getSeconds, t.getNanos)
- .`with`(ChronoField.ERA, era)
- // Add days separately to convert dates existed in Julian calendar but
not
- // in Proleptic Gregorian calendar. For example, 1000-02-29 is valid date
- // in Julian calendar because 1000 is a leap year but 1000 is not a leap
- // year in Proleptic Gregorian calendar. And 1000-02-29 doesn't exist in
it.
- .plusDays(t.getDate - 1) // Returns the next valid date after
`date.getDate - 1` days
- instantToMicros(localDateTime.atZone(ZoneId.systemDefault).toInstant)
+ val micros = millisToMicros(t.getTime) + (t.getNanos / NANOS_PER_MICROS) %
MICROS_PER_MILLIS
Review comment:
This seems to be missing in `branch-3.0`.
```
$ git grep millisToMicros
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala:
val micros = millisToMicros(t.getTime) + (t.getNanos / NANOS_PER_MICROS) %
MICROS_PER_MILLIS
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
