Re: [SQL] Reconciling Beam SQL Environments with Calcite Schema

Reviews are wrapping up, this will probably merge Monday if I don't hear from anyone else. One more TableProvider API change after review feedback: getTables now returns Map instead of Set. Andrew On Thu, May 3, 2018 at 10:41 AM Andrew Pilloud wrote: > Ok,

Re: [SQL] Reconciling Beam SQL Environments with Calcite Schema

Ok, I've finished with this change. Didn't get reviews on the early cleanup PRs, so I've pushed all these changes into the first cleanup PR: https://github.com/apache/beam/pull/5224 Andrew On Tue, May 1, 2018 at 10:35 AM Andrew Pilloud wrote: > I'm just starting to move

Re: [SQL] Reconciling Beam SQL Environments with Calcite Schema

I'm just starting to move forward on this. Looking at my team's short term needs for SQL, option one would be good enough, however I agree with Kenn that we want something like option two eventually. I also don't want to break existing users and it sounds like there is at least one custom

Re: [SQL] Reconciling Beam SQL Environments with Calcite Schema

Kenn: yes, MetaStore is user-facing, Users can choose to implement their own MetaStore, currently only an InMemory implementation in Beam CodeBase. Andrew: I like the second option, since it "retains the ability for DDL operations to be processed by a custom MetaStore.", IMO we should have the

Re: [SQL] Reconciling Beam SQL Environments with Calcite Schema

Can you say more about how the metastore is used? I presume it is or will be user-facing, so are Beam SQL users already providing their own? I'm sure we want something like that eventually to support things like Apache Atlas and HCatalog, IIUC for the "create if needed" logic when using Beam SQL

[SQL] Reconciling Beam SQL Environments with Calcite Schema

I'm working on updating our Beam DDL code to use the DDL execution functionality that recently merged into core calcite. This enables us to take advantage of Calcite JDBC as a way to use Beam SQL. As part of that I need to reconcile the Beam SQL Environments with the Calcite Schema (which is