[
https://issues.apache.org/jira/browse/FLINK-41524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated FLINK-41524:
-------------------------------
Affects Version/s: 2.1.3
1.20.5
1.19.2
2.0.3
2.2.3
2.3.2
> Filter pushdown rewrites a nested field access on a function result
> (f(x).a.b) into a wrong reference
> -----------------------------------------------------------------------------------------------------
>
> Key: FLINK-41524
> URL: https://issues.apache.org/jira/browse/FLINK-41524
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.19.2, 1.20.5, 2.0.3, 2.1.3, 2.2.3, 2.3.2
> Reporter: Chen Zhang
> Priority: Major
>
> When a WHERE predicate accesses a field two or more levels deep on the result
> of a function call, e.g. {{{}COALESCE(r1, r2).a.b = 'x'{}}}, and the table
> source implements {{{}SupportsFilterPushDown{}}}, the planner corrupts the
> predicate. The function call and the field-access chain are dropped, and the
> predicate becomes {{{}r1 = 'x'{}}}. That predicate is:
> * passed to {{{}applyFilters(){}}},
> * kept as the remaining filter in the {{Calc}} above the scan,
> * and used by projection pushdown, which prunes the other arguments
> ({{{}r2{}}}).
> The query silently returns wrong results. Disabling
> {{table.optimizer.source.predicate-pushdown-enabled}} gives correct results.
> h3. Steps to reproduce
> Data file {{{}/tmp/repro/part-0.json{}}}:
> {noformat}
> {"r1":{"a":{"b":"x"}},"r2":{"a":{"b":"y"}},"s":"s1"}
> {"r1":null,"r2":{"a":{"b":"x"}},"s":"s2"}
> {noformat}
> {code:sql}
> CREATE TEMPORARY TABLE t (
> r1 ROW<a ROW<b STRING>>,
> r2 ROW<a ROW<b STRING>>,
> s STRING
> ) WITH ('connector'='filesystem', 'path'='file:///tmp/repro',
> 'format'='json');
> SELECT s FROM t WHERE COALESCE(r1, r2).a.b = 'x';
> {code}
> *Expected:* {{{}s1{}}}, {{s2}}
> *Actual:* 0 rows
> Plan with pushdown enabled (default):
> {noformat}
> Calc(select=[s], where=[(r1 = 'x')])
> +- TableSourceScan(table=[[default_catalog, default_database, t,
> filter=[=(r1, _UTF-16LE'x')], project=[r1, s], metadata=[]]], fields=[r1, s])
> {noformat}
> Plan with {{{}SET 'table.optimizer.source.predicate-pushdown-enabled' =
> 'false';{}}}:
> {noformat}
> Calc(select=[s], where=[(COALESCE(r1, r2).a.b = 'x')])
> +- TableSourceScan(table=[[default_catalog, default_database, t]],
> fields=[r1, r2, s])
> {noformat}
> h3. Root cause
> In {{RexNodeToExpressionConverter.visitFieldAccess}}
> ({{{}RexNodeExtractor.scala{}}}):
> {code:scala}
> fieldAccess.getReferenceExpr match {
> // push down on nested field inside a composite type like map or array is
> not supported
> case _: RexCall => return None
> case _ =>
> }
> val schema =
> NestedProjectionUtil.build(Collections.singletonList(fieldAccess), dataType)
> val fieldIndices = NestedProjectionUtil.convertToIndexArray(schema)
> var (topLevelColumnName, nestedColumn) = schema.columns.head
> ...
> {code}
> # *The guard only checks the immediate base.* For {{{}COALESCE(r1,
> r2).a.b{}}}, the outer access's base is the field access {{{}COALESCE(r1,
> r2).a{}}}, not a {{{}RexCall{}}}, so the guard passes.
> # *{{NestedProjectionUtil.build}} fails without saying so.* In
> {{{}NestedSchemaExtractor.visitFieldAccess{}}}, {{internalVisit}} reaches the
> {{RexCall}} ({{{}case expr => expr.accept(this){}}}). That registers the
> call's operands ({{{}r1{}}}, {{{}r2{}}}) as top-level leaf columns, then
> returns {{success = false}} and exits early. {{build()}} returns that schema
> anyway.
> # *The converter trusts the schema.* It takes {{schema.columns.head}}
> ({{{}r1{}}}) and builds a {{{}NestedFieldReferenceExpression(["r1"]){}}}. It
> gives it the type of the original field access ({{{}STRING{}}}), so the
> corrupted predicate still passes type checking.
> # {*}The corrupted expression becomes the remaining filter{*}, and {{r2}} is
> pruned from the scan's columns.
> A single level of access ({{{}f(x).a{}}}) is caught by the existing guard;
> two or more levels are not.
> h3. Proposed fix
> Follow the chain of field accesses down to its base, and don't convert unless
> that base is a {{{}RexInputRef{}}}:
> {code:scala}
> @tailrec
> private def baseOf(e: RexNode): RexNode = e match {
> case f: RexFieldAccess => baseOf(f.getReferenceExpr)
> case other => other
> }
> override def visitFieldAccess(fieldAccess: RexFieldAccess):
> Option[ResolvedExpression] = {
> if (!baseOf(fieldAccess).isInstanceOf[RexInputRef]) {
> return None
> }
> ...
> }
> {code}
> Optionally, {{NestedSchemaExtractor}} could also stop registering a call's
> operands as leaf columns when called from this path, or the converter could
> check {{{}build(){}}}'s result before using {{{}schema.columns.head{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)