EnricoMi commented on pull request #27805:
URL: https://github.com/apache/spark/pull/27805#issuecomment-640604977
@srowen CalendarIntervals are defined by three values: microseconds, days
and months. These are what I refered to as "resolutions" above. You cannot
convert between either of them without ambiguity. This is what @yaooqinn meant
with "contract".
To be unambiguous, division here is only defined when both intervals define
only microseconds or days or months, no mixture. CalendarIntervals encodes
microseconds, seconds, minutes and hours as microseconds, days and weeks as
days, and months and years as months. So the division allows to divide a
CalendarInterval of, lets say 1h with 5m, or 2h and 30m with 30s, or 8 days
with 1 week, or 2 years with 2 months. As long as the division is unambiguous.
@yaooqinn can I use the mentioned Java API in `Dataset.select(...)` or
`org.apache.spark.sql.functions.expr`?
With your suggested `date_part`,
.withColumn("duration in hours", $"interval" / lit("1
hour").cast(CalendarIntervalType))
becomes
.withColumn("duration in hours", date_part("hour", $"interval") +
date_part("minute", $"interval") / 60.0 + date_part("second", $"interval") /
3600.0 + date_part("microsecond", $"interval") / 3600000000.0)
which is not very readable.
All I want to achieve here is get access to the magnitude of an interval.
CalendarInterval is neither comparable, nor convertable in a number
representing the whole of the interval (and not just parts like `date_part`
does). The mentioned `extractAsDuration` and `extractAsPeriod` might be the
closest to achieve that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
