Re: Migrating Beam SQL to Calcite's code generation

I've spent most of the day getting beam working on Calcite 1.18 and now I'm just starting to work on making your Joda UDF changes compatible with Calcite's code generation, so I haven't gotten very far into investigating how we might handle this yet. Calcite has multiple aggregate function

Re: Migrating Beam SQL to Calcite's code generation

For the not working SESSION_END(), I had an investigation on it: https://issues.apache.org/jira/browse/BEAM-5799 https://issues.apache.org/jira/browse/CALCITE-2645 According to the reply in the Calcite JIRA, there might be some other way to implement SESSION_END. I haven't looked into it though.

Re: Migrating Beam SQL to Calcite's code generation

1. Window start/end: Actually this is already provided in other ways and the window in the SQL environment is unused and just waiting to be deleted. So you can still access TUMBLE_START, etc. This is well-defined as a part of the row so there's no semantic problem, but I think it should already

Re: Migrating Beam SQL to Calcite's code generation

>From https://issues.apache.org/jira/browse/BEAM-5204 it seems like what you most care about is to have joins that trigger more than once per window. To accomplish it you hope to build an "escape hatch" from SQL/relational semantics to specialized Beam SQL semantics. It could make sense with

Re: Migrating Beam SQL to Calcite's code generation

Raise this thread. Seems there're more changes in the backend on how a FUNCTION is executed in the backend, as noticed in #6996 : 1. BeamSqlExpression and BeamSqlExpressionExecutor are removed; 2. BeamSqlExpressionEnvironment are removed; Then, 1. for

Re: Migrating Beam SQL to Calcite's code generation

This is a so exciting change! Since we cannot mix current implementation with Calcite code generation, is there any case that Calcite code generation does not support but our current implementation supports, so switching to Calcite code generation will have some impact to existing usage? -Rui

Re: Migrating Beam SQL to Calcite's code generation

To follow up on this, the PR is now in a reviewable state and I've added more tests for FLOOR and CEIL. Both work with a more extensive set of arguments after this change. There are now 4 outstanding calcite PRs that get all the tests passing. Unfortunately there is no easy way to mix our current

Re: Migrating Beam SQL to Calcite's code generation

Thats Awesome! Thanks Team! On Tue, Sep 18, 2018 at 10:58 AM Jean-Baptiste Onofré wrote: > Awesome, > > thanks Andrew !! > > Regards > JB > > On 17/09/2018 23:27, Andrew Pilloud wrote: > > I've adapted Calcite's EnumerableCalc code generation to generate the > > BeamCalc DoFn. The primary

Re: Migrating Beam SQL to Calcite's code generation

Awesome, thanks Andrew !! Regards JB On 17/09/2018 23:27, Andrew Pilloud wrote: > I've adapted Calcite's EnumerableCalc code generation to generate the > BeamCalc DoFn. The primary purpose behind this change is so we can take > advantage of Calcite's extensive SQL operator implementation. This

Re: Migrating Beam SQL to Calcite's code generation

That seems pretty amazing. On Mon, Sep 17, 2018 at 3:22 PM Mingmin Xu wrote: > Awesome work, we should call Calcite operator functions if available. > > I haven't get time to read the PR yet, for those impacted would keep > existing implementation. One example is, I notice FLOOR/CEIL only

Re: Migrating Beam SQL to Calcite's code generation

Awesome work, we should call Calcite operator functions if available. I haven't get time to read the PR yet, for those impacted would keep existing implementation. One example is, I notice FLOOR/CEIL only supports months/years recently which is quite a surprise to me. Mingmin On Mon, Sep 17,

Re: Migrating Beam SQL to Calcite's code generation

This is pretty amazing! Thank you for doing this! Regards, Anton On Mon, Sep 17, 2018 at 2:27 PM Andrew Pilloud wrote: > I've adapted Calcite's EnumerableCalc code generation to generate the > BeamCalc DoFn. The primary purpose behind this change is so we can take > advantage of Calcite's

Migrating Beam SQL to Calcite's code generation

I've adapted Calcite's EnumerableCalc code generation to generate the BeamCalc DoFn. The primary purpose behind this change is so we can take advantage of Calcite's extensive SQL operator implementation. This deletes ~11000 lines of code from Beam (with ~350 added), significantly increases the set