[
https://issues.apache.org/jira/browse/CALCITE-7687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Pelissier updated CALCITE-7687:
---------------------------------------
Description:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(), // [2]
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// [1]
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
{{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate)}} method:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
// [1]
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
both currently pass down the predicate to their input [1] without considering
any possible translation, since an {{Aggregate}} derives its row type as
{{(group keys..., aggregate calls...)}} and so its output field {{i}} is input
field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes the
predicate, it can end up reading a different column from the one the predicate
names.
The consequence is silent rather than an exception, like the tests attached to
the first two comments, where after
{{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
{{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table exposes
through {{RelOptTable#unwrap}}, which reports the null fraction of a field ($1)
that is not the one the predicate names. This $1 is the Aggregate's second
group key, which is field ($2) of the scan underneath it.
Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
very closely related to this ticket.
Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
uses {{RelOptUtil.pushPastProject}} and
{{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
(which "Converts an expression that is based on the output fields of a Calc to
an equivalent expression on the Calc's input fields.") to convert the predicate
before passing it to the input. There is no equivalent
{{RelOptUtil.pushPastAggregate}}, even though {{FilterAggregateTransposeRule}}
already computes that same mapping inline in order to push a {{Filter}} past an
{{Aggregate}}.
was:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(), // [2]
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// [1]
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
and
{{{}[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate){}}}:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
// [1]
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
both currently pass down the predicate to their input [1] without considering
any possible translation, since an {{Aggregate}} derives its row type as
{{(group keys..., aggregate calls...)}} and so its output field {{i}} is input
field {{{}groupSet.nth(i\){}}}; hence when the Aggregate's input analyzes the
predicate, it can end up reading a different column from the one the predicate
names. Note that in the second method the group key beside it *is* translated,
by {{{}RelMdUtil.setAggChildKeys{}}}.
Unlike CALCITE-4414 the consequence is silent rather than an exception:
{{splitFilters}} only pushes conjuncts whose references are contained in the
bitmap it is given, so the index that reaches the input is always valid, it
simply names another field. The estimate does not change either as long as the
input keys only off {{{}SqlKind{}}}, which stock {{RelMdUtil.guessSelectivity}}
does. It becomes a wrong estimate for a table that supplies a
{{BuiltInMetadata.Selectivity.Handler}} or a
{{BuiltInMetadata.DistinctRowCount.Handler}} through
{{{}RelOptTable#unwrap{}}}, the hook added by CALCITE-4223 so that engines can
derive selectivity from column statistics. In the tests attached to the first
two comments such a handler is asked about column {{c}} and answers with column
{{{}b{}}}'s null fraction.
Note I: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
uses {{RelOptUtil.pushPastProject}} and
{{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
(added by CALCITE-4414) to convert the predicate before passing it to the
input. There is no equivalent {{{}RelOptUtil.pushPastAggregate{}}}, which is
plausibly why these two overloads were missed.
Note II: in the first code snippet above, {{splitFilters}} is called at [2]
with {{{}rel.getGroupSet(){}}}, which holds *input* indices, while
{{predicate}} is expressed over the {*}output{*}; {{RelMdDistinctRowCount}}
passes {{ImmutableBitSet.range(rel.getGroupCount())}} for the same purpose, so
the two disagree. A predicate on a genuine group key can therefore be refused,
and a predicate on an aggregate call can be pushed — in the second test {{IS
NULL($2)}} on {{COUNT($0)}} is answered with a column's null fraction.
> RelMdSelectivity and RelMdDistinctRowCount for Aggregate can propagate a
> predicate with wrong references
> --------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7687
> URL: https://issues.apache.org/jira/browse/CALCITE-7687
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.42.0
> Reporter: Etienne Pelissier
> Assignee: Etienne Pelissier
> Priority: Minor
> Labels: in-progress, pull-request-available
>
> {{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
> rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
> {code:java}
> public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
> @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> rel.getGroupSet(), // [2]
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPred =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> // [1]
> Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
> ...
> }
> {code}
> {{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
> rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
> predicate)}} method:
> {code:java}
> public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
> mq,
> ImmutableBitSet groupKey, @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> ImmutableBitSet.range(rel.getGroupCount()),
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPreds =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> // set the bits as they correspond to the child input
> ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
> RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
> // [1]
> Double distinctRowCount =
> mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
> ...
> }
> {code}
> both currently pass down the predicate to their input [1] without considering
> any possible translation, since an {{Aggregate}} derives its row type as
> {{(group keys..., aggregate calls...)}} and so its output field {{i}} is
> input field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes
> the predicate, it can end up reading a different column from the one the
> predicate names.
> The consequence is silent rather than an exception, like the tests attached
> to the first two comments, where after
> {{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
> {{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
> predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table
> exposes through {{RelOptTable#unwrap}}, which reports the null fraction of a
> field ($1) that is not the one the predicate names. This $1 is the
> Aggregate's second group key, which is field ($2) of the scan underneath it.
> Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
> very closely related to this ticket.
> Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
> uses {{RelOptUtil.pushPastProject}} and
> {{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
> (which "Converts an expression that is based on the output fields of a Calc
> to an equivalent expression on the Calc's input fields.") to convert the
> predicate before passing it to the input. There is no equivalent
> {{RelOptUtil.pushPastAggregate}}, even though
> {{FilterAggregateTransposeRule}} already computes that same mapping inline in
> order to push a {{Filter}} past an {{Aggregate}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
