Hi Walaa and Tao: I am very happy to see your sharing. Our team is also working on SQL rewriting, analysis and optimization. Using Calcite's materialized view recognition capabilities to speed up user queries, many materialized view recognition algorithms[1] and normalization algorithms[2] have been implemented, and a lot of work has been done in preprocessing available materialized views, which has excellent performance. Welcome to discuss issues related to materialized view identification.
[1] https://github.com/apache/calcite/pull/2094 [2] https://github.com/apache/calcite/pull/2262 Regards! Zhaohui Xu ------------------ ???????? ------------------ ??????: "dev" <[email protected]>; ????????: 2020??12??12??(??????) ????9:23 ??????: "dev"<[email protected]>; ????: Re: Using Calcite at LinkedIn Hi Walaa Very happy to see this, our team basically do the same thing, a unified SQL layer: 1. Spark: RelNode -> Spark DataFrame plan 2. Presto: RelNode -> In string SQL 3. Clickhouse: RelNode -> Serialized RelNode 4. Flink -> TBD(with datastream API or table API) I do point 1 both in my previous company and current company, maybe I can participate in this part: analyze and translate Spark Catalyst plans. Regards! Aron Tao Walaa Eldin Moustafa <[email protected]> ??2020??12??12?????? ????5:34?????? > Hi Calcite community, > > I wanted to share a recently published LinkedIn's blog series article [1] > on how Calcite helps us build a smarter data lake using Coral [2]. Hope you > find it interesting. Also, if you want to discuss with our team and the > data lake + Calcite community, please feel free to join our Coral Slack > workspace [3]. > > [1] https://engineering.linkedin.com/blog/2020/coral > [2] https://github.com/linkedin/coral > [3] > > https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g > > Thanks, > Walaa. >
