carloea2 commented on code in PR #8506:
URL: https://github.com/apache/texera/pull/8506#discussion_r4357232052
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/intervalJoin/IntervalJoinOpDesc.scala:
##########
@@ -168,3 +224,47 @@ class IntervalJoinOpDesc extends LogicalOp {
}
}
+
+object IntervalJoinOpDesc {
+
+ /** The upper bound of each interval, as the executor computes it:
+ * `Timestamp.valueOf(t.toLocalDateTime.plusX(constant))`. The constant is
+ * added on the machine's wall clock, so a day across a daylight-saving
+ * change is 23 or 25 hours, and a month counts from the local date. A wall
+ * clock that falls in a gap or an overlap is read at the standard offset,
+ * as java.util.Date reads it, and one before 1900 at the standard offset
+ * today. Each distinct key is worked out once, since the column holds every
+ * pair of the cross join.
+ */
+ val IntervalUpper: String =
+ """def _texera_interval_upper(s, offset):
+ | from dateutil.tz import tzlocal
+ |
+ | zone = _texera_local_zone()
+ | standard = _texera_standard_offset(zone)
+ | moved = _texera_local_wall_clocks(s.dt.tz_localize("UTC")) + offset
Review Comment:
Source timestamps from #8597 are already naive local wall clocks, so
labeling them UTC and converting them again changes the interval. In
America/Chicago, right=2024-03-09 12:00 with a one-day interval matches
left=2024-03-10 11:30 natively but drops it here. Across November's clock
change, the export adds the 12:30 row that native rejects. UTC and
ordinary-date controls pass. Please use the source wall-clock representation
and cover these inputs; the current test strips Z from UTC instants instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
