Re: SQL Windowing

Thanks for the quick reply Brian! I've filed a JIRA for option (a): https://jira.apache.org/jira/browse/BEAM-10143 Makes sense to define DATETIME as a logical type. I'll check out your PR. We could work around this for now by doing a cast, e.g.: TUMBLE(CAST(f_timestamp AS DATETIME), INTERVAL

Re: SQL Windowing

Hey Max, Thanks for kicking the tires on SqlTransform in Python :) We don't have any tests of windowing and Sql in Python yet, so I'm not that surprised you're running into issues here. Portable schemas don't support the DATETIME type, because we decided not to define it as one of the atomic

SQL Windowing

Hi, I'm using the SqlTransform as an external transform from within a Python pipeline. The SQL docs [1] mention that you can either (a) window the input or (b) window in the SQL query. Option (a): input | "Window >> beam.WindowInto(window.FixedWindows(30)) | "Aggregate" >>

Re: [SQL] Windowing and triggering changes proposal

I'm late to the party as usual, but also added some comments. Overall supportive of this work. Thanks for the clear analysis, Anton. On Tue, Jan 16, 2018 at 10:58 AM Mingmin Xu wrote: > Thanks @Anton for the proposal. Window(w/ trigger) support in SQL is > limited now,

Re: [SQL] Windowing and triggering changes proposal

Thanks @Anton for the proposal. Window(w/ trigger) support in SQL is limited now, you're very welcome to join the improvement. There's a balance between injected DSL mode and CLI mode when we were implementing BealmSQL overall, not only widowing. Many default behaviors are introduced to make it

Re: [SQL] Windowing and triggering changes proposal

I've commented on the doc. This is a really nice analysis and I think the proposal is good for making SQL work with Beam windowing and triggering in a way that will make sense to users. Kenn On Thu, Jan 11, 2018 at 4:05 PM, Anton Kedin wrote: > Hi, > > Wanted to gather

[SQL] Windowing and triggering changes proposal

Hi, Wanted to gather feedback on changes I propose to the behavior of some aspects of windowing and triggering in Beam SQL. In short: Beam SQL currently overrides input PCollections' windowing/triggering configuration in few cases. For example if a query has a simple GROUP BY clause, we would