Hello Julian/Calcite community,
I am working on adding subquery support in HIVE using calcite. From what I
have read/understood so far Calcite requires HIVE to create RexSubqueryNode
corresponding to a subquery and then call SubQueryRemoveRule to get rid of
RexSubqueryNode and change it to join. This seems to be working for
Un-correlated queries where SubQueryRemoveRule creates Aggregate + Join to get
rid of RexSubqueryNode. But I am running into following issues with Co-rrelated
queries: (Note that I am using FILTER rule)
* Looking at SubQueryRemoveRule code it should be creating Correlate node
if it finds any correlation in given filter. To find if given filter has
correlation getVariablesSet is called on filter, which supposedly should be
returning set of correlated variables, but it is always returning empty set as
filter does not implement this method. Shouldn’t Filter implement this method
to return appropriate correlated variables ?
* Comments in SubQueryRemoveRule mentions that “The correlate can be
removed using RelDecorrelator”. But I don’t see SubqueryRemoveRule using
RelDecorrelator to de-correlate given query. Should SubQueryRemoveRule call
this ? If not is doing de-correlation immediately after SubQueryRemoveRule
Here is what I have done so far for co-rrelated queries. Could you please
comment if this is right ?
* While creating RexSubqueryNode and RelNode for the subquery I am creating
RexCorrelVariable. RexCorrelVariable needs a correlation id. CorrelationId
requires an integer id. Should this id be same as index of co-relatted column
in outer table ?
* Hive has a HiveFilter which is extended from Filter. I implemented
getVariableSet method to look at the condition and return all correlated
variables in condition’s RelNode. Does this sound correct ?
* I am calling RelDecorrelator’s decorrelateQuery immediately after calling
SubQueryRemoveRule. After implementing getVariableSet in HiveFilter
SubQueryRemoveRule seems to be creating appropriate LogicalCorrelate for
correlate queries but decorrelateQuery is throwing an exception.