We have a handler implemented, for example we have implemented getPrivacyMode() method in the RelMetadataQuery as a proof of concept and it seems to do the right thing. Our question was regarding how to read the underlying data from the source database. We can open a JDBC connection and read in the data for the metadata provider to use - however we were not sure if there was existing functionality to bring in table metadata.
If, for example, we wanted to make a metadata provider which provides histograms for cardinality estimates, where/how should the provider read in the histograms from the underlying database? On Tue, Nov 19, 2019 at 1:00 PM Julian Hyde <[email protected]> wrote: > > is there existing functionality to read in external > > statistics and metadata - for example things like table statistics. > > Yes. Write a handler for that kind of metadata, and put it in the > RelMetadataProvider that Calcite uses to prepare the query. Then when > it asks for (say) the row count of a table it will call your handler > rather than the built-in one. > > On Tue, Nov 19, 2019 at 6:08 AM Madhav Suresh <[email protected]> > wrote: > > > > Public/private columns are attributes we created for our implementation > of > > a private query processor. Right now we store those attributes in > > information_schema. We want to read those into calcite for Query planning > > > > Madhav > > > > On Tue, Nov 19, 2019 at 4:47 AM Muhammad Gelbana <[email protected]> > > wrote: > > > > > To me, your question is still unclear. Are you asking about system > columns > > > [1] ? > > > > > > What are public/private attributes ? > > > > > > [1] https://www.postgresql.org/docs/current/ddl-system-columns.html > > > > > > > > > On Tue, Nov 19, 2019 at 5:12 AM Madhav Suresh < > [email protected]> > > > wrote: > > > > > > > Thanks for the reply Julian! > > > > > > > > I should post more details: right now we have a Metadata class which > > > > implements the Metadata spec. All of that seems to work fine. Where > we > > > are > > > > stuck is in how to read the information_schema metadata from JDBC > into > > > the > > > > planner. One hackish solution is to open a DB connection when our > > > Metadata > > > > getter is called on TableScan, and run a query against the underlying > > > > database. However, is there existing functionality to read in > external > > > > statistics and metadata - for example things like table statistics. > > > > > > > > Madhav > > > > > > > > On Mon, Nov 18, 2019 at 8:42 PM Julian Hyde <[email protected]> > wrote: > > > > > > > > > May need to be a new type of metadata (i.e. a class that implements > > > > > Metadata and is acquired via RelMetadataQuery) [1] [2]. > > > > > > > > > > Julian > > > > > > > > > > [1] > > > > > > > > > > > > > https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/Metadata.html > > > > > < > > > > > > > > > > > > > https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/Metadata.html > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > > https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/package-summary.html > > > > > < > > > > > > > > > > > > > https://calcite.apache.org/apidocs/org/apache/calcite/rel/metadata/package-summary.html > > > > > > > > > > > > > > > > > > > > > On Nov 18, 2019, at 5:40 PM, Madhav Suresh < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi All, > > > > > > > > > > > > We have security type information that we want to bring into the > > > > > optimizer. > > > > > > Consider the query: > > > > > > SELECT a FROM a,b where a.id=b.id AND a.p > 10; > > > > > > We want the optimizer to know that a.id, b.id are "public" > > > attributes, > > > > > and > > > > > > a.p is a private attribute. Currently we have type information > stored > > > > > > inform information_schema in postgres. Right now our prototype > > > > considers > > > > > > privacy as a RelTrait. We are stuck on what the best way to read > this > > > > > > information is. Is there a similar example, or template on how to > > > read > > > > in > > > > > > table metadata into calcite? > > > > > > > > > > > > Madhav > > > > > > > > > > > > > > > > > >
