Hi Julian, On Tue, Sep 10, 2019 at 5:19 PM Julian Feinauer < [email protected]> wrote:
> > when going through the Code I just had another Idea. > Currently a TableFunction is executed as EnumerableTableFunctionScan which > gets generated from a LogicalTableFunctionScan by the Rule > EnumerableTableFunctionScanRule. > What if you just remove that Rule and add a custom Rule of yours which > translates it of a TableScan of your taste? > > That is indeed something I had thought about. In reference to your earlier question, yes, there are parameters needed for the TableFunction, so I think that the DrillTable approach wouldn't work in my case. I didn't see any easy way to alter rules outside of RelNode.register, but I assume that that is indeed possible, so I'll look further into this approach. Thanks for the advice. - Gabriel > Julian > > Am 10.09.19, 08:13 schrieb "Julian Feinauer" < > [email protected]>: > > Hi Gabriel, > > thats an interesting question for me too. > Do you need parameters for those "dynamic tables"? > If not you could do it similar to what Drill is doing and just > implement a Schema which always returns "true" if someone asks for a Table > and then returns a Table Implementation that you provide where you can hook > in later and add the functionality that you in fact need. This can then > also be used in optimization as you can then control your custom Table type. > Perhaps it helps to look at the DrillTable class in [1]. > > On a Side node I try to figure out what would be necessary to make > TableFunction wo also work with TranslatableTable. > Would you mind opening an issue in Jira for that? > > Julian > > [1] > https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java > > Am 10.09.19, 03:25 schrieb "Gabriel Reid" <[email protected]>: > > Hi, > > I'm currently using a combination of TableFunctions and > TableMacros to > expose various dynamic (relatively unstructured) data sources via > Calcite. > The underlying data sources are such that data can only be > retrieved by > first specifying what you want (i.e. there is no catalog of all > data that > is available). > > I'm currently handling this by using a combination of > TableFunctions and > TableMacros. > > The issue that I'm running into comes when I want to implement > custom > planner rules for the underlying functionality. As far I as I can > see, it's > not possible to register planner rules based on a > TableFunctionImpl, > because a TableFunctionImpl only exposes a ScannableTable, so > there's no > chance to hook into RelOptNode.register. > > On the other hand, implementing a TableMacro does allow to return a > TranslatableTable, which then does allow intercepting the call to > RelOptNode.register to register rules. However, TableMacros > require that > all parameters are literals, and I'm providing timestamps, via > TIMESTAMPADD() and CURRENT_TIMESTAMP() calls, which then doesn't > work for > TableMacros (all parameters to a table macro need to be literals, > otherwise > query validation fails in Calcite). > > I'm wondering if I'm missing some built-in functionality which > would make > it possible to have a dynamic table function/macro that can also be > manipulated via custom planner rules. > > Options (which may or may not exist) that I can think of are: > * something that would/could visit all macro parameters ahead of > time and > resolve things like CURRENT_TIMESTAMP() to a literal, before > further query > validation occurs > * register rules somewhere outside of RelOptNode.register (e.g. > when the > schema is first created) > > Are there any currently-working options in Calcite that can help > me do what > I'm trying to do? And if there aren't and I would add such a thing > to > Calcite, are there any suggestions as to what the most appropriate > approach > would be (either one of the two options I listed above, or > something else)? > > Thanks, > > Gabriel > > > > >
