Re: Towards a spec for robust streaming SQL, Part 1

Tyler, Yes, dynamic table changes over time. You can find more details about dynamic table from this Flink blog ( https://flink.apache.org/news/2017/04/04/dynamic-tables.html). Fabian, me and Xiaowei posted it a week before the flink-forward@SF. "A dynamic table is a table that is continuously

Re: Towards a spec for robust streaming SQL, Part 1

Being able to support an EMIT config independent of the query itself sounds great for compatible use cases (which should be many :-). Shaoxuan, can you please refresh my memory what a dynamic table means in Flink? It's basically just a state table, right? The "dynamic" part of the name is to

Re: Towards a spec for robust streaming SQL, Part 1

Thanks to Tyler and Fabian for sharing your thoughts. Regarding to the early/late update control of FLINK. IMO, each dynamic table can have an EMIT config. For FLINK table-API, this can be easily implemented in different manners, case by case. For instance, in window aggregate, we could define

Re: Towards a spec for robust streaming SQL, Part 1

On Tue, May 9, 2017 at 3:06 PM Fabian Hueske wrote: > Hi Tyler, > > thank you very much for this excellent write-up and the super nice > visualizations! > You are discussing a lot of the things that we have been thinking about as > well from a different perspective. > IMO,

Re: Towards a spec for robust streaming SQL, Part 1

Hi Tyler, thank you very much for this excellent write-up and the super nice visualizations! You are discussing a lot of the things that we have been thinking about as well from a different perspective. IMO, yours and the Flink model are pretty much aligned although we use a different terminology

Re: Towards a spec for robust streaming SQL, Part 1

No worries, thanks for the heads up. Good luck wrapping all that stuff up. -Tyler On Tue, Apr 25, 2017 at 12:07 AM Fabian Hueske wrote: > Hi Tyler, > > thanks for pushing this effort and including the Flink list. > I haven't managed to read the doc yet, but just wanted to

Re: Towards a spec for robust streaming SQL, Part 1

Hi Tyler, thanks for pushing this effort and including the Flink list. I haven't managed to read the doc yet, but just wanted to thank you for the write-up and let you know that I'm very interested in this discussion. We are very close to the feature freeze of Flink 1.3 and I'm quite busy

Re: Towards a spec for robust streaming SQL, Part 1

Good point, when you start talking about anything less than a full join, triggers get involved to describe how one actually achieves the desired semantics, and they may end up being tied to just one of the inputs (e.g., you may only care about the watermark for one side of the join). Am expecting

Re: Towards a spec for robust streaming SQL, Part 1

There's something to be said about having different triggering depending on which side of a join data comes from, perhaps? (delightful doc, as usual) Kenn On Fri, Apr 21, 2017 at 1:33 PM, Tyler Akidau wrote: > Thanks for reading, Luke. The simple answer is that

Re: Towards a spec for robust streaming SQL, Part 1

Thanks for reading, Luke. The simple answer is that CoGBK is basically flatten + GBK. Flatten is a non-grouping operation that merges the input streams into a single output stream. GBK then groups the data within that single union stream as you might otherwise expect, yielding a single table. So I

Re: Towards a spec for robust streaming SQL, Part 1

The doc is a good read. I think you do a great job of explaining table -> stream, stream -> stream, and stream -> table when there is only one stream. But when there are multiple streams reading/writing to a table, how does that impact what occurs? For example, with CoGBK you have multiple

Towards a spec for robust streaming SQL, Part 1

Hello Beam, Calcite, and Flink dev lists! Apologies for the big cross post, but I thought this might be something all three communities would find relevant. Beam is finally making progress on a SQL DSL utilizing Calcite, thanks to Mingmin Xu. As you can imagine, we need to come to some