MaxGekk commented on a change in pull request #28205: [SPARK-31439][SQL] Fix
perf regression of fromJavaDate
URL: https://github.com/apache/spark/pull/28205#discussion_r407905485
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -101,16 +101,10 @@ object DateTimeUtils {
* @return The number of days since epoch from java.sql.Date.
*/
def fromJavaDate(date: Date): SQLDate = {
- val era = if (date.before(julianCommonEraStart)) 0 else 1
- val localDate = LocalDate
- .of(date.getYear + 1900, date.getMonth + 1, 1)
- .`with`(ChronoField.ERA, era)
- // Add days separately to convert dates existed in Julian calendar but
not
- // in Proleptic Gregorian calendar. For example, 1000-02-29 is valid date
- // in Julian calendar because 1000 is a leap year but 1000 is not a leap
- // year in Proleptic Gregorian calendar. And 1000-02-29 doesn't exist in
it.
- .plusDays(date.getDate - 1) // Returns the next valid date after
`date.getDate - 1` days
- localDateToDays(localDate)
+ val millisUtc = date.getTime
+ val millisLocal = millisUtc + TimeZone.getDefault.getOffset(millisUtc)
+ val julianDays = Math.toIntExact(Math.floorDiv(millisLocal,
MILLIS_PER_DAY))
Review comment:
@cloud-fan Spark 2.4 uses floating point ops, see
https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L137
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
