Hi there, It's exciting that all of us have the same idea, Calcite to Beam, and our team already made it's true for the last few months, including tumbling/hopping window newly released in Calcite 1.12.0, and we will release in our product environment at the end of this month if no big delay.
We want to share our work to the community, and make Calcite SQL to Beam more universal and powerful with you. We made some extension， as follows: 1) add "create table" grammer 2) add "create UDF/UDTF/UDAF" grammer 3) use "view" to support multi-line sql in one DAG 4) add some functions not exist in Calcite now( https://issues.apache.org/jira/browse/CALCITE-1733) 5) the table operators for SQL on beam 6) ... for example: -- DDL for udafCREATE UDAF avg_test(DOUBLE) RETURNS DOUBLE AS 'com.test.AvgTest'; -- DDL for stream input CREATE STREAM TABLE source_1( rt TIMESTAMP, f1 BIGINT, f2 DOUBLE, f3 VARCHAR, TIMESTAMP by rt -- assign rt column as rowtime) WITH ( type='socket', port='9999'); -- DDL for view with group & filter resultCREATE VIEW view_1 (rt, f3, f2, f1) AS SELECT HOP_END(rt, INTERVAL '1' SECOND, INTERVAL '5' SECOND) AS hop_rt, avg_test(f3) AS avg_f3, f2 AS f2, f1 AS f1FROM source_1 GROUP BY HOP(rt, INTERVAL '1' SECOND, INTERVAL '5' SECOND), f1, f2WHERE f1 % 2 = 1; -- DDL for dimension table (which used as a static table when join)CREATE DIM TABLE dim_table( `key` BIGINT, `value` VARCHAR) WITH ( type='mysql', jdbcUrl='jdbc:mysql://127.0.0.1:3306/test', tableName='test', username='test', password='test'); -- DDL for join resultCREATE VIEW view_2 (rt, f1, f2, f3, f4) AS SELECT t1.rt AS rt, t1.f1 AS f1, t1.f2 AS f2, t1.f3 AS f3, t2.value AS f4FROM view_1 AS t1JOIN dim_table AS t2 ON t1.f1 = t2.key -- DDL for result 1CREATE RESULT TABLE result_1( console_rt TIMESTAMP, console_f1 BIGINT, console_f2 DOUBLE, console_f3 VARCHAR, console_f4 VARCHAR) WITH ( type='console'); -- DDL for result 2CREATE RESULT TABLE result_2( mysql_rt TIMESTAMP, mysql_f1 BIGINT, mysql_f2 DOUBLE,) WITH ( type='mysql', ...); -- DML for result_1INSERT INTO result_1SELECT *FROM view_2WHERE f2 > 3; -- DML for result_2INSERT INTO result_2SELECT rt, f1, f2FROM view_2WHERE f2 <= 3; 2017-04-06 7:02 GMT+08:00 Mingmin Xu <[email protected]>: > We have some work already, will create a feature branch later soon. > > > > On Wed, Apr 5, 2017 at 3:37 PM, Khai Tran <[email protected]> > wrote: > > > Thanks, Julian. That's what I need. > > > > On Wed, Apr 5, 2017 at 3:26 PM, Julian Hyde <[email protected]> wrote: > > > > > + dev@beam (please moderate through!) > > > > > > There’s only been talk so far. > > > > > > I found https://issues.apache.org/jira/browse/BEAM-301 < > > > https://issues.apache.org/jira/browse/BEAM-301>. Does that match what > > you > > > were thinking? If so, let’s continue the conversation there. > > > > > > Julian > > > > > > > On Apr 5, 2017, at 3:13 PM, Khai Tran <[email protected]> > > > wrote: > > > > > > > > Hi, > > > > Just want to check if there is any ongoing effort to convert Calcite > > > > logical plan into Apache Beam APIs for stream processing? > > > > > > > > Thanks, > > > > Khai > > > > > > > > > > > > -- > ---- > Mingmin >
