At FlinkForward the SQL BoF (birds-of-a-feather) group discussed several issues
relating to streaming SQL, but the discussion of stream-to-table joins was
particularly interesting. It raises issues of time semantics if the table is
changing over time, especially if we wish to be able to re-play the stream
later, join to the table, and get the same results.
I have been thinking about the problem for the past few days, and have written
up my thoughts in a document . The link is read-only, but I welcome
comments, so contact me back-channel if you would like write access.
It is a rather lengthy read, I am afraid, partly because I include a lot of
examples. But I believe that the new concept that I have introduced, the
temporal table function (which I often abbreviate temporal table) has huge
expressive power, and can be a foundation for the semantics of streaming query
systems that need to interact with “historic” data.
Please give it a read, and comment either in this email thread or in the
document. Also, please feel free to forward it to others outside the Calcite
community who are working on streaming query semantics, and invite them to