MaxGekk commented on a change in pull request #24311: [SPARK-27401][SQL]
Refactoring conversion of Timestamp to/from java.sql.Timestamp
URL: https://github.com/apache/spark/pull/24311#discussion_r272820174
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralGenerator.scala
##########
@@ -102,8 +102,14 @@ object LiteralGenerator {
lazy val dateLiteralGen: Gen[Literal] =
for { d <- Arbitrary.arbInt.arbitrary } yield Literal.create(new Date(d),
DateType)
- lazy val timestampLiteralGen: Gen[Literal] =
- for { t <- Arbitrary.arbLong.arbitrary } yield Literal.create(new
Timestamp(t), TimestampType)
+ private def yearToMillis(year: Long): Long = {
+ year * 365 * 24 * 3600 * 1000L
+ }
+
+ lazy val timestampLiteralGen: Gen[Literal] = {
+ for { millis <- Gen.choose(yearToMillis(-9999), yearToMillis(10000)) }
+ yield Literal.create(new Timestamp(millis), TimestampType)
+ }
Review comment:
I changed the random generator for timestamps, and limit max/min values for
milliseconds since epoch because the long random generator can produce
milliseconds that caused Long overflow at conversion of milliseconds to
microseconds. Internally as you know we store microseconds since epoch in
`TimestampType`. For example, the old (current) generator can create an
instance of `java.sql.Timestamp(-3948373668011580000, 570000000)`. New function
`fromJavaTimestamp` calls `instantToMicros`, and inside of it we use
`multiplyExact` which can detect `Long` overflow on multiplication:
```scala
def instantToMicros(instant: Instant): Long = {
val us = Math.multiplyExact(-3948373668011580, 1000000)
...
}
```
<img width="873" alt="Screen Shot 2019-04-07 at 08 41 30"
src="https://user-images.githubusercontent.com/1580697/55679740-fa961780-5910-11e9-914c-11a1336894e8.png";>
Previous (current) implementation doesn't detect the overflow at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/1580697/55679740-fa961780-5910-11e9-914c-11a1336894e8.png"](https://user-images.githubusercontent.com/1580697/55679740-fa961780-5910-11e9-914c-11a1336894e8.png"){kind=link}