I do agree that Apache Beam can represent dates and times with arbitrary
precision and can do it many different ways.
My argument has always been should around whether we restrict this range to
a common standard to increase interoperability across other systems. For
example, SQL database servers have varying degrees as to what ranges they
support:
* Oracle 10[1]: 0001-01-01 to 9999-12-31
* Oracle 11g[2]: Julian era, ranging from January 1, 4712 BCE through
December 31, 9999 CE (Common Era, or 'AD'). Unless BCE ('BC' in the format
mask)
* MySQL[3]: '1000-01-01 00:00:00' to '9999-12-31 23:59:59'
* Microsoft SQL: January 1, 1753, through December 31, 9999 for
datetime[4] and January 1,1 CE through December 31, 9999 CE for datetime2[5]
The common case of the global window containing timestamps that are before
and after all of these supported ranges above means that our users can't
represent a global window within a database using its common data types.
1: https://docs.oracle.com/javadb/10.8.3.0/ref/rrefdttlimits.html
2:
https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#CNCPT413
3: https://dev.mysql.com/doc/refman/8.0/en/datetime.html
4:
https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime-transact-sql?view=sql-server-ver15
5:
https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql?view=sql-server-ver15
On Wed, Nov 13, 2019 at 3:28 AM Jan Lukavský <[email protected]> wrote:
> Hi,
>
> just an idea on these related topics that appear these days - it might
> help to realize, that what we actually don't need a full arithmetic on
> timestamps (Beam model IMHO doesn't need to know exactly what is the exact
> difference of two events). What we actually need is a slightly simplified
> algebra. Given two timestamps T1 and T2 and a "duration" (a different type
> from timestamp), we need operations (not 100% sure that this is exhaustive,
> but seems to be):
>
> - is_preceding(T1, T2): bool
>
> - important !is_preceding(T1, T2) does NOT imply that is_preceding(T2,
> T1) - !is_preceding(T1, T2) && !is_preceding(T2, T1) would mean events are
> _concurrent_
>
> - this relation has to be also antisymmetric
>
> - given this function we can construct a comparator, where multiple
> distinct timestamps can be "equal" (or with no particular ordering, which
> is natural property of time)
>
> - min_timestamp_following(T1, duration): T2
>
> - that would return a timestamp for which is_preceding(T1 + duration,
> T2) would return true and no other timestamp X would exist for which
> is_preceding(T1 + duration, X) && is_preceding(X, T2) would be true
>
> - actually, this function would serve as the definition for the
> duration object
>
> If we can supply this algebra, it seems that we can use any representation
> of timestamps and intervals. It might be (probably) even possible to let
> user specify his own type used as timestamps and durations, which could
> solve the issues of not currently being able to correctly represent
> timestamps lower than Long.MIN_VALUE (although we can get data for that low
> timestamps - cosmic microwave background being one example :)). Specifying
> this algebra actually probably boils down to proposal (3) in Robert's
> thread [1].
>
> Just my 2 cents.
>
> Jan
>
> [1]
> https://lists.apache.org/thread.html/1672898393cb0d54a77a879be0fb5725902289a3e5063d0f9ec36fe1@%3Cdev.beam.apache.org%3E
> On 11/13/19 10:11 AM, jincheng sun wrote:
>
> Thanks for bringing up this discussion @Luke.
>
> As @Kenn mentioned, in Beam we have defined the constants value for the
> min/max/end of global window. I noticed that
> google.protobuf.Timestamp/Duration is only used in window definitions,
> such as FixedWindowsPayload, SlidingWindowsPayload, SessionsPayload, etc.
>
> I think that both RFC 3339 and Beam's current implementation are big
> enough to express a common window definitions. But users can really
> define a window size that outside the scope of the RFC 3339. Conceptually,
> we should not limit the time range for window(although I think the range of
> RPC 3339 is big enough in most cases).
>
> To ensure that people well know the background of the discussion, hope you
> don't mind that I put the original conversion thread[1] here.
>
> Best,
> Jincheng
>
> [1] https://github.com/apache/beam/pull/10041#discussion_r344380809
>
> Robert Bradshaw <[email protected]> 于2019年11月12日周二 下午4:09写道：
>
>> I agree about it being a tagged union in the model (together with
>> actual_time(...) - epsilon). It's not just a performance hack though,
>> it's also (as discussed elsewhere) a question of being able to find an
>> embedding into existing datetime libraries. The real question here is
>> whether we should limit ourselves to just these 10000 years AD, or
>> find value in being able to process events for the lifetime of the
>> universe (or, at least, recorded human history). Artificially limiting
>> in this way would seem surprising to me at least.
>>
>> On Mon, Nov 11, 2019 at 11:58 PM Kenneth Knowles <[email protected]> wrote:
>> >
>> > The max timestamp, min timestamp, and end of the global window are all
>> performance hacks in my view. Timestamps in beam are really a tagged union:
>> >
>> > timestamp ::= min | max | end_of_global | actual_time(... some
>> quantitative timestamp ...)
>> >
>> > with the ordering
>> >
>> > min < actual_time(...) < end_of_global < max
>> >
>> > We chose arbitrary numbers so that we could do simple numeric
>> comparisons and arithmetic.
>> >
>> > Kenn
>> >
>> > On Mon, Nov 11, 2019 at 2:03 PM Luke Cwik <[email protected]> wrote:
>> >>
>> >> While crites@ was investigating using protobuf to represent Apache
>> Beam timestamps within the TestStreamEvents, he found out that the well
>> known type google.protobuf.Timestamp doesn't support certain timestamps we
>> were using in our tests (specifically the max timestamp that Apache Beam
>> supports).
>> >>
>> >> This lead me to investigate and the well known type
>> google.protobuf.Timestamp supports dates/times from 0001-01-01T00:00:00Z to
>> 9999-12-31T23:59:59.999999999Z which is much smaller than the timestamp
>> range that Apache Beam currently supports -9223372036854775ms to
>> 9223372036854775ms which is about 292277BC to 294247AD (it was difficult to
>> find a time range that represented this).
>> >>
>> >> Similarly the google.protobuf.Duration represents any time range over
>> those ~10000 years. Google decided to limit their range to be compatible
>> with the RFC 3339[2] standard to which does simplify many things since it
>> guarantees that all RFC 3339 time parsing/manipulation libraries are
>> supported.
>> >>
>> >> Should we:
>> >> A) define our own timestamp/duration types to be able to represent the
>> full time range that Apache Beam can express?
>> >> B) limit the valid timestamps in Apache Beam to some standard such as
>> RFC 3339?
>> >>
>> >> This discussion is somewhat related to the efforts to support nano
>> timestamps[2].
>> >>
>> >> 1: https://tools.ietf.org/html/rfc3339
>> >> 2:
>> https://lists.apache.org/thread.html/86a4dcabdaa1dd93c9a55d16ee51edcff6266eda05221acbf9cf666d@%3Cdev.beam.apache.org%3E
>>
>
