At FlinkForward the SQL BoF (birds-of-a-feather) group discussed several issues relating to streaming SQL, but the discussion of stream-to-table joins was particularly interesting. It raises issues of time semantics if the table is changing over time, especially if we wish to be able to re-play the stream later, join to the table, and get the same results.
I have been thinking about the problem for the past few days, and have written up my thoughts in a document [1]. The link is read-only, but I welcome comments, so contact me back-channel if you would like write access. It is a rather lengthy read, I am afraid, partly because I include a lot of examples. But I believe that the new concept that I have introduced, the temporal table function (which I often abbreviate temporal table) has huge expressive power, and can be a foundation for the semantics of streaming query systems that need to interact with “historic” data. Please give it a read, and comment either in this email thread or in the document. Also, please feel free to forward it to others outside the Calcite community who are working on streaming query semantics, and invite them to participate also. Julian https://docs.google.com/document/d/1RvnLEEQK92axdAaZ9XIU5szpkbGqFMBtzYiIY4dHe0Q/edit?usp=sharing
