Thanks a lot for writing this up. And sorry for taking so long to respond
on this; attending Strata and then catching up after Strata ended up being
more time consuming than I'd hoped. :-P
Overall, I love the temporal table idea. I think it's a very clean way of
capturing and accessing the evolution of a table over time. I also like how
the example used concisely motivates the challenges involved.
I actually don't have any major comments for you at this time. I think the
idea is solid at a high level (and I'm generally not qualified to comment
on the nitty gritty SQL details :-).
My only real complaint is that, as you allude to in the Further Work
section, it doesn't address the details of handling out-of-order data. But
after thinking about it a bit, I believe it meshes nicely with the
semantics of the EMIT statement, at least as I was envisioning them (still
not 100% sure we're in agreement there yet, but maybe). I was not expecting
the two to interact so cleanly when I first started reading your doc, so I
find that encouraging.
I'm going to try to write up my thoughts on this over the next week or two
with some diagrams to show what I mean. I'll post the doc on this thread
once it's done.
On Wed, Sep 21, 2016 at 6:57 PM Julian Hyde <jh...@apache.org> wrote:
> At FlinkForward the SQL BoF (birds-of-a-feather) group discussed several
> issues relating to streaming SQL, but the discussion of stream-to-table
> joins was particularly interesting. It raises issues of time semantics if
> the table is changing over time, especially if we wish to be able to
> re-play the stream later, join to the table, and get the same results.
> I have been thinking about the problem for the past few days, and have
> written up my thoughts in a document . The link is read-only, but I
> welcome comments, so contact me back-channel if you would like write access.
> It is a rather lengthy read, I am afraid, partly because I include a lot
> of examples. But I believe that the new concept that I have introduced, the
> temporal table function (which I often abbreviate temporal table) has huge
> expressive power, and can be a foundation for the semantics of streaming
> query systems that need to interact with “historic” data.
> Please give it a read, and comment either in this email thread or in the
> document. Also, please feel free to forward it to others outside the
> Calcite community who are working on streaming query semantics, and invite
> them to participate also.