Can you say more about how the metastore is used? I presume it is or will be user-facing, so are Beam SQL users already providing their own?
I'm sure we want something like that eventually to support things like Apache Atlas and HCatalog, IIUC for the "create if needed" logic when using Beam SQL to create a derived data set. But I don't think we should build out those code paths until we have at least one non-in-memory implementation. Just a really high level $0.02. Kenn On Mon, Apr 23, 2018 at 4:56 PM Andrew Pilloud <apill...@google.com> wrote: > I'm working on updating our Beam DDL code to use the DDL execution > functionality that recently merged into core calcite. This enables us to > take advantage of Calcite JDBC as a way to use Beam SQL. As part of that I > need to reconcile the Beam SQL Environments with the Calcite Schema (which > is calcite's environment). We currently have copies of our tables in the > Beam meta/store, Calcite Schema, BeamSqlEnv, and BeamQueryPlanner. I have a > pending PR which merges the later two to just use the Calcite Schema copy. > Merging the Beam MetaStore and Calcite Schema isn't as simple. I have two > options I'm looking for feedback on: > > 1. Make Calcite Schema authoritative and demote MetaStore to be something > more like a Calcite TableFactory. Calcite Schema already implements the > semantics of our InMemoryMetaStore. If the Store interface is just over > built, this approach would result in a significant reduction in code. This > would however eliminate the CRUD part of the interface leaving just the > buildBeamSqlTable function. > > 2. Pass the Beam MetaStore into Calcite wrapped with a class translating > to Calcite Schema (like we do already with tables). Instead of copying > tables into the Calcite Schema we would pass in Beam meta/store as the > source of truth and Calcite would manipulate tables directly in the Beam > meta/store. This is a bit more complicated but retains the ability for DDL > operations to be processed by a custom MetaStore. > > Thoughts? > > Andrew >
