Hey,
when going through the Code I just had another Idea.
Currently a TableFunction is executed as EnumerableTableFunctionScan which gets
generated from a LogicalTableFunctionScan by the Rule
EnumerableTableFunctionScanRule.
What if you just remove that Rule and add a custom Rule of yours which
translates it of a TableScan of your taste?
Julian
Am 10.09.19, 08:13 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>:
Hi Gabriel,
thats an interesting question for me too.
Do you need parameters for those "dynamic tables"?
If not you could do it similar to what Drill is doing and just implement a
Schema which always returns "true" if someone asks for a Table and then returns
a Table Implementation that you provide where you can hook in later and add the
functionality that you in fact need. This can then also be used in optimization
as you can then control your custom Table type.
Perhaps it helps to look at the DrillTable class in [1].
On a Side node I try to figure out what would be necessary to make
TableFunction wo also work with TranslatableTable.
Would you mind opening an issue in Jira for that?
Julian
[1]
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java
Am 10.09.19, 03:25 schrieb "Gabriel Reid" <gabriel.r...@gmail.com>:
Hi,
I'm currently using a combination of TableFunctions and TableMacros to
expose various dynamic (relatively unstructured) data sources via
Calcite.
The underlying data sources are such that data can only be retrieved by
first specifying what you want (i.e. there is no catalog of all data
that
is available).
I'm currently handling this by using a combination of TableFunctions and
TableMacros.
The issue that I'm running into comes when I want to implement custom
planner rules for the underlying functionality. As far I as I can see,
it's
not possible to register planner rules based on a TableFunctionImpl,
because a TableFunctionImpl only exposes a ScannableTable, so there's no
chance to hook into RelOptNode.register.
On the other hand, implementing a TableMacro does allow to return a
TranslatableTable, which then does allow intercepting the call to
RelOptNode.register to register rules. However, TableMacros require that
all parameters are literals, and I'm providing timestamps, via
TIMESTAMPADD() and CURRENT_TIMESTAMP() calls, which then doesn't work
for
TableMacros (all parameters to a table macro need to be literals,
otherwise
query validation fails in Calcite).
I'm wondering if I'm missing some built-in functionality which would
make
it possible to have a dynamic table function/macro that can also be
manipulated via custom planner rules.
Options (which may or may not exist) that I can think of are:
* something that would/could visit all macro parameters ahead of time
and
resolve things like CURRENT_TIMESTAMP() to a literal, before further
query
validation occurs
* register rules somewhere outside of RelOptNode.register (e.g. when the
schema is first created)
Are there any currently-working options in Calcite that can help me do
what
I'm trying to do? And if there aren't and I would add such a thing to
Calcite, are there any suggestions as to what the most appropriate
approach
would be (either one of the two options I listed above, or something
else)?
Thanks,
Gabriel
