AngersZhuuuu commented on a change in pull request #32945:
URL: https://github.com/apache/spark/pull/32945#discussion_r654117032
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala
##########
@@ -154,7 +156,15 @@ class RandomDataGeneratorSuite extends SparkFunSuite with
SQLHelper {
val data = generator.apply()
val catalyst = toCatalyst(data)
val convertedBack = toScala(catalyst)
- assert(data == convertedBack)
+ dt match {
Review comment:
> shall we update the `RandomDataGenerator`? to make sure we generate
correct data for `interval year`.
Thinking this way too, should change `RandomDataGenerator` side since it
will impact many test.
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGeneratorSuite.scala
##########
@@ -154,7 +156,15 @@ class RandomDataGeneratorSuite extends SparkFunSuite with
SQLHelper {
val data = generator.apply()
val catalyst = toCatalyst(data)
val convertedBack = toScala(catalyst)
- assert(data == convertedBack)
+ dt match {
Review comment:
updated
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
##########
@@ -284,7 +285,10 @@ object RandomDataGenerator {
new CalendarInterval(months, days, ns)
})
case _: DayTimeIntervalType => Some(() => Duration.of(rand.nextLong(),
ChronoUnit.MICROS))
- case _: YearMonthIntervalType => Some(() =>
Period.ofMonths(rand.nextInt()).normalized())
+ case YearMonthIntervalType(_, YEAR) =>
+ val period = Period.ofMonths(rand.nextInt()).normalized()
+ Some(() => Period.ofYears(period.getYears).normalized())
Review comment:
> And I guess that `.normalized()` is not needed since we don't have
other fields - so nothing to normalize.
normalize is for checking equal `P-1155869325M did not equal
P-96322443Y-9M`
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
##########
@@ -284,7 +285,10 @@ object RandomDataGenerator {
new CalendarInterval(months, days, ns)
})
case _: DayTimeIntervalType => Some(() => Duration.of(rand.nextLong(),
ChronoUnit.MICROS))
- case _: YearMonthIntervalType => Some(() =>
Period.ofMonths(rand.nextInt()).normalized())
+ case YearMonthIntervalType(_, YEAR) =>
+ val period = Period.ofMonths(rand.nextInt()).normalized()
+ Some(() => Period.ofYears(period.getYears).normalized())
Review comment:
> can we just do `Period.ofYears(rand.nextInt() / 12).normalized())`?
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
