uros-b commented on code in PR #58855:
URL: https://github.com/apache/spark/pull/58855#discussion_r4027021333
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:
##########
@@ -4981,8 +4981,13 @@ case class ConvertTimezone(
- "SECOND"
- "MILLISECOND"
- "MICROSECOND"
+ - "NANOSECOND" - only for nanosecond-precision timestamp inputs
(TIMESTAMP_NTZ(p) /
+ TIMESTAMP_LTZ(p), p in [7, 9]); the result is floored to the
input's precision, so a
+ quantity finer than the type's step (10^(9-p) ns) is truncated to
it
* quantity - this is the number of units of time that you want to add.
- * timestamp - this is a timestamp (w/ or w/o timezone) to which you want
to add.
+ * timestamp - this is a timestamp (w/ or w/o timezone) to which you want
to add. A
+ nanosecond-precision timestamp keeps its sub-microsecond fraction;
units of MICROSECOND or
+ coarser leave the fraction unchanged.
""",
Review Comment:
PySpark timestamp_add still documents units as ending at MICROSECOND. SQL
ExpressionDescription was updated; the Python list was not.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala:
##########
@@ -991,6 +991,61 @@ object DateTimeUtils extends SparkDateTimeUtils {
}
}
+ /**
+ * Adds the specified number of units to a nanosecond-precision timestamp of
precision `p` (in
+ * [7, 9]), returning a value floored to that precision.
+ *
+ * For units of MICROSECOND or coarser the addition runs on the microsecond
grid (reusing
+ * [[timestampAdd]] for all the calendar, DST and overflow handling) and the
input's already
+ * `p`-aligned fraction is carried through unchanged. For the NANOSECOND
unit the fraction absorbs
+ * `quantity` nanoseconds and any whole microseconds carry into
`epochMicros`; the resulting
+ * fraction is then floored to `p` (via
[[truncateTimestampNanosToPrecision]]) so a NANOSECOND
+ * quantity finer than the type's step never produces an off-grid value that
would compare, hash
+ * or sort unequal to its displayed (aligned) form. NANOSECOND is only valid
here: adding
+ * nanoseconds to a microsecond timestamp is unrepresentable, so that
combination is rejected as
+ * an invalid unit through the microsecond [[timestampAdd]] path.
+ *
+ * @param unit A keyword that specifies the interval units to add to the
input timestamp.
+ * @param quantity The amount of `unit`s to add. It can be positive or
negative.
+ * @param ts The input nanosecond-precision timestamp.
+ * @param precision The declared fractional-second precision `p` in [7, 9]
of the input/output.
+ * @param zoneId The time zone ID at which the operation is performed.
+ * @return A nanosecond-precision timestamp value floored to `precision`.
+ */
+ def timestampAddNanos(
+ unit: String,
+ quantity: Long,
+ ts: TimestampNanosVal,
+ precision: Int,
+ zoneId: ZoneId): TimestampNanosVal = {
+ if (unit.toUpperCase(Locale.ROOT) == "NANOSECOND") {
+ try {
+ // Split the added nanoseconds into whole microseconds and a [0, 999]
remainder first, so
+ // the fraction sum stays within [0, 1998] and only the true
microsecond total can overflow
+ // a Long (adding the remainder to `nanosWithinMicro` before the split
would spuriously
+ // reject a large-but-representable quantity).
+ val quotientMicros = Math.floorDiv(quantity, NANOS_PER_MICROS)
+ val remainderNanos = Math.floorMod(quantity, NANOS_PER_MICROS)
+ val fractionSum = ts.nanosWithinMicro.toLong + remainderNanos
+ val carryMicros =
+ Math.addExact(quotientMicros, Math.floorDiv(fractionSum,
NANOS_PER_MICROS))
+ val newFraction = Math.floorMod(fractionSum, NANOS_PER_MICROS).toShort
+ val newMicros = Math.addExact(ts.epochMicros, carryMicros)
+ truncateTimestampNanosToPrecision(
+ TimestampNanosVal.fromParts(newMicros, newFraction), precision)
+ } catch {
+ case _: ArithmeticException | _: DateTimeException =>
+ throw QueryExecutionErrors.timestampAddOverflowError(ts.epochMicros,
quantity, unit)
Review Comment:
Overflow still goes through timestampAddOverflowError, which renders only
epochMicros. Pre-existing helper; the fraction is dropped from the error text.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
