Hi Gabriel, thats an interesting question for me too. Do you need parameters for those "dynamic tables"? If not you could do it similar to what Drill is doing and just implement a Schema which always returns "true" if someone asks for a Table and then returns a Table Implementation that you provide where you can hook in later and add the functionality that you in fact need. This can then also be used in optimization as you can then control your custom Table type. Perhaps it helps to look at the DrillTable class in [1].
On a Side node I try to figure out what would be necessary to make TableFunction wo also work with TranslatableTable. Would you mind opening an issue in Jira for that? Julian [1] https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java Am 10.09.19, 03:25 schrieb "Gabriel Reid" <gabriel.r...@gmail.com>: Hi, I'm currently using a combination of TableFunctions and TableMacros to expose various dynamic (relatively unstructured) data sources via Calcite. The underlying data sources are such that data can only be retrieved by first specifying what you want (i.e. there is no catalog of all data that is available). I'm currently handling this by using a combination of TableFunctions and TableMacros. The issue that I'm running into comes when I want to implement custom planner rules for the underlying functionality. As far I as I can see, it's not possible to register planner rules based on a TableFunctionImpl, because a TableFunctionImpl only exposes a ScannableTable, so there's no chance to hook into RelOptNode.register. On the other hand, implementing a TableMacro does allow to return a TranslatableTable, which then does allow intercepting the call to RelOptNode.register to register rules. However, TableMacros require that all parameters are literals, and I'm providing timestamps, via TIMESTAMPADD() and CURRENT_TIMESTAMP() calls, which then doesn't work for TableMacros (all parameters to a table macro need to be literals, otherwise query validation fails in Calcite). I'm wondering if I'm missing some built-in functionality which would make it possible to have a dynamic table function/macro that can also be manipulated via custom planner rules. Options (which may or may not exist) that I can think of are: * something that would/could visit all macro parameters ahead of time and resolve things like CURRENT_TIMESTAMP() to a literal, before further query validation occurs * register rules somewhere outside of RelOptNode.register (e.g. when the schema is first created) Are there any currently-working options in Calcite that can help me do what I'm trying to do? And if there aren't and I would add such a thing to Calcite, are there any suggestions as to what the most appropriate approach would be (either one of the two options I listed above, or something else)? Thanks, Gabriel
