I've seen that in many adapters, although they implement the TranslatableTable interface, Queryable is still used (see the usage of table.getExpression(...) in XToEnumerableConverter for Elasticsearch, Mongo, and Geode, to name a few). What is the reasoning there?
Il giorno mar 25 gen 2022 alle ore 20:15 Julian Hyde <[email protected]> ha scritto: > +1 what Stamatis said. Queryable is for compatibility with LINQ. If you > want to build an adapter that supports push-down, you will likely use > FilterableTable for simple adapters, TranslatableTable for more complex > adapters. In neither case will you need to deal with Queryable. > > Stamatis laid out best practices in his excellent BOSS tutorial: > https://www.youtube.com/watch?v=meI0W12f_nw < > https://www.youtube.com/watch?v=meI0W12f_nw>. (I am co-presenter in other > parts of the tutorial, but all credit for the adapters material goes to > Stamatis.) > > Julian > > > > > On Jan 23, 2022, at 12:46 PM, Stamatis Zampetakis <[email protected]> > wrote: > > > > Hi Julian, > > > > I don't think there is an easy way to understand the Queryable interface > > unless you check how it is used. I don't see it as something that you > need > > to implement no matter what but more as a convenience API that will > > facilitate the integration with the Enumerable operators (if you rely on > > them). Even in that case it could be possible to skip it. > > > > There are many ways to push a filter down into the underlying engine and > I > > think the Calcite code base has already quite a few examples on how this > > can be done (JdbcAdapter, ElasticSearch, Druid, etc). There is no one > > option that is best in all cases. Using one you may (e.g., > FilterableTable) > > write more concise code and using another (e.g., custom rules + custom > > operators) may lead to more powerful optimizations or an easier > extensible > > system. > > > > Regarding materialized views there are many built in things in Calcite. > The > > best place to start would probably be the website [1] > > > > The bindable convention uses interpretation most of the time but it also > > involves code generation in some parts (e.g., BindableFilter). The > > Enumerable convention is more widely used in the wild so I would say it > is > > a more stable and better option to begin with. Afterwards you may need to > > invest in building in-house operators to solve some invonveniences of > > Calcite built-in conventions. > > > > Best, > > Stamatis > > > > [1] https://calcite.apache.org/docs/materialized_views.html > > > > On Fri, Jan 21, 2022 at 1:21 PM Julian Feinauer < > > [email protected]> wrote: > > > >> Hi all, > >> > >> in the last weeks I worked on Integrating the Apache IoTDB Project with > >> Calcite. > >> This covers two possible scenarios. One, to use Apache IoTDB as an > Adapter > >> in Apache Calcite (like MongoDB, Cassandra, et al) and on the other > hand we > >> are looking at using Calcites Query Optimizer to introduce indexing into > >> the IoTDB server (the IoTDB Server builds a RelNode / Tree and passes > it to > >> the planner, after planning the resulting RelNode is then processed > further > >> by the IoTDB Server, executed and returned). > >> > >> I looked a lot on the other Adapters and how they are implemented and > have > >> some questions: > >> > >> One rather general question is about the Queryable<> Interface. I tried > to > >> look up all the docs (also in Linq) but still not fully understand it. > From > >> my understanding it is like a Enumerable<> but it has a “native” way to > >> already to things like ordering or filtering. So if I have a Queryable<> > >> which implements a custom Filter an automated “Push Down” can be done by > >> the framework without a Rule or code generation. > >> > >> One important requirement for us in IoTDB is to do the query pushdown to > >> the TableScan (which is done implicitly in the current server but is > first > >> explicit in the RelNode that we generate). > >> So whats the best way to “merge” a LogicalFilter and a IoTDBTableScan > to a > >> “filtered” scan? > >> Is the right way to return a QueryableTable as TableScan and the Planner > >> will take care by generating the call to ‘.filter(…)’. > >> The same applies to ordering. > >> > >> Another question that is important for us is the usage of “Materialized > >> Views” or other “Indexes”. > >> As we handle basically always timeseries in most cases the only suitable > >> index is a “Materialized View” on parts of the time series which we can > use > >> to replace parts of the Relational Tree to avoid IO and computation for > >> parts that are already precomputed. > >> > >> Is there already an existing support for that in Calcite or would we > just > >> write custom Rules for our cases? > >> > >> My last question is about the Callable TraitDef. So far I only used > >> Enumerable Convention which results in Code generation (which has an > >> impact on the query latency). Am I right in assuming that the Binable > >> Convention is somehow similar to the Enumerable Convention with the only > >> difference that it does not do code generation but interpretation? > >> And to potentially use both (depending on whatever switch we set) we > just > >> have to provide Converter Rules for both? > >> What would you use in a Server setup? Always Enumerable? > >> > >> Thanks already for any responses or hints! > >> Julian F > >> > >
