MrPowers commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757523412
@yaooqinn - thanks for commenting. It's generally easier to work with
functions/classes that take Column arguments instead of Integer / Long
arguments.
Suppose you have a DataFrame with a `TimestampType` column called
`some_time` and a `LongType` column called `some_millis`. I don't think there
is an easy way to build a `CalendarInterval` object from the `some_millis`
column and add it to the `some_time` column. This does not work:
```scala
df.withColumn("new_time", col("d") + new CalendarInterval(0, 0,
col("milliseconds")))
```
Other datetime functions have also had this problem in the past. In Spark
1.5 the `date_add` function was added with this method signature: `def
date_add(start: Column, days: Int): Column`. In Spark 3, this method signature
was added to make `date_add` more usable: `def date_add(start: Column, days:
Column): Column`, see [this
JIRA](https://issues.apache.org/jira/browse/SPARK-28598). It's easy to go from
a Scala object to a Column object with `lit`. It's not easy to go from a
Column object to a Scala object.
`CalendarInterval` is also a bit more complicated for less technical users.
They need to run `import org.apache.spark.unsafe.types.CalendarInterval` and
need to learn how to instantiate Java classes. They also need to figure out
how to convert hours / minutes to microseconds. Think `CalendarInterval` is a
great abstraction and is useful in a lot of contexts, but think
`make_interval()` will be easier for most users.
Let me know if any of these arguments convince you that `make_interval()` is
a good idea ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
