Peng-Lei commented on a change in pull request #32999:
URL: https://github.com/apache/spark/pull/32999#discussion_r655824124
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
##########
@@ -2790,7 +2790,9 @@ class ColumnExpressionSuite extends QueryTest with
SharedSparkSession {
).foreach { case (end, start) =>
val df = Seq((end, start)).toDF("end", "start")
val daysBetween = Duration.ofDays(ChronoUnit.DAYS.between(start,
end))
- checkAnswer(df.select($"end" - $"start"), Row(daysBetween))
+ val r = df.select($"end" - $"start").toDF("diff")
+ checkAnswer(r, Row(daysBetween))
+ assert(r.schema === new StructType().add("diff",
DayTimeIntervalType(0, 0)))
Review comment:
> could you add tests for them too
@sarutak ,Sorry I am wrong. First I should exclude the `Date +/- Interval`
from this PR. Second Even if I wanted to change `Date +/- Interval`, I should
convert the interval to the number of days `ExtractANSIIntervalDays)` and then
use `DateAdd` expression as cloud-fan said.
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -349,8 +349,18 @@ class Analyzer(override val catalogManager: CatalogManager)
case p: LogicalPlan => p.transformExpressionsUpWithPruning(
_.containsPattern(BINARY_ARITHMETIC), ruleId) {
case a @ Add(l, r, f) if a.childrenResolved => (l.dataType,
r.dataType) match {
- case (DateType, _: DayTimeIntervalType) => TimeAdd(Cast(l,
TimestampType), r)
- case (_: DayTimeIntervalType, DateType) => TimeAdd(Cast(r,
TimestampType), l)
+ case (DateType, dit: DayTimeIntervalType) =>
Review comment:
> I agree with change but I think it's a separate thing and can be done
in a new PR.
>
> The impl detail can also be improved: we can convert the interval to the
number of days (`ExtractANSIIntervalDays`) and then use `DateAdd` expression.
@cloud-fan Could I propose a ticket for improved for `DateType +/-
DayTimeIntervalType(DAY)` and try to resolve it ?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -2701,8 +2702,11 @@ case class SubtractDates(
override def inputTypes: Seq[AbstractDataType] = Seq(DateType, DateType)
override def dataType: DataType = {
- // TODO(SPARK-35727): Return INTERVAL DAY from dates subtraction
- if (legacyInterval) CalendarIntervalType else DayTimeIntervalType()
+ if (legacyInterval) {
+ CalendarIntervalType
+ } else {
+ DayTimeIntervalType(DAY)
+ }
Review comment:
@MaxGekk Thank you very much, I fix it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
