[ 
https://issues.apache.org/jira/browse/FLINK-41524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-41524:
-----------------------------------
    Labels: pull-request-available  (was: )

> Filter pushdown rewrites a nested field access on a function result 
> (f(x).a.b) into a wrong reference
> -----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-41524
>                 URL: https://issues.apache.org/jira/browse/FLINK-41524
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Planner
>    Affects Versions: 1.19.2, 1.20.5, 2.0.3, 2.1.3, 2.2.3, 2.3.2
>            Reporter: Chen Zhang
>            Priority: Major
>              Labels: pull-request-available
>
> When a WHERE predicate accesses a field two or more levels deep on the result 
> of a function call, e.g. {{{}COALESCE(r1, r2).a.b = 'x'{}}}, and the table 
> source implements {{{}SupportsFilterPushDown{}}}, the planner corrupts the 
> predicate. The function call and the field-access chain are dropped, and the 
> predicate becomes {{{}r1 = 'x'{}}}. That predicate is:
>  * passed to {{{}applyFilters(){}}},
>  * kept as the remaining filter in the {{Calc}} above the scan,
>  * and used by projection pushdown, which prunes the other arguments 
> ({{{}r2{}}}).
> The query silently returns wrong results. Disabling 
> {{table.optimizer.source.predicate-pushdown-enabled}} gives correct results.
> h3. Steps to reproduce
> Data file {{{}/tmp/repro/part-0.json{}}}:
> {noformat}
> {"r1":{"a":{"b":"x"}},"r2":{"a":{"b":"y"}},"s":"s1"}
> {"r1":null,"r2":{"a":{"b":"x"}},"s":"s2"}
> {noformat}
> {code:sql}
> CREATE TEMPORARY TABLE t (
>   r1 ROW<a ROW<b STRING>>,
>   r2 ROW<a ROW<b STRING>>,
>   s  STRING
> ) WITH ('connector'='filesystem', 'path'='file:///tmp/repro', 
> 'format'='json');
> SELECT s FROM t WHERE COALESCE(r1, r2).a.b = 'x';
> {code}
> *Expected:* {{{}s1{}}}, {{s2}}
> *Actual:* 0 rows
> Plan with pushdown enabled (default):
> {noformat}
> Calc(select=[s], where=[(r1 = 'x')])
> +- TableSourceScan(table=[[default_catalog, default_database, t, 
> filter=[=(r1, _UTF-16LE'x')], project=[r1, s], metadata=[]]], fields=[r1, s])
> {noformat}
> Plan with {{{}SET 'table.optimizer.source.predicate-pushdown-enabled' = 
> 'false';{}}}:
> {noformat}
> Calc(select=[s], where=[(COALESCE(r1, r2).a.b = 'x')])
> +- TableSourceScan(table=[[default_catalog, default_database, t]], 
> fields=[r1, r2, s])
> {noformat}
> h3. Root cause
> In {{RexNodeToExpressionConverter.visitFieldAccess}} 
> ({{{}RexNodeExtractor.scala{}}}):
> {code:scala}
> fieldAccess.getReferenceExpr match {
>   // push down on nested field inside a composite type like map or array is 
> not supported
>   case _: RexCall => return None
>   case _ =>
> }
> val schema = 
> NestedProjectionUtil.build(Collections.singletonList(fieldAccess), dataType)
> val fieldIndices = NestedProjectionUtil.convertToIndexArray(schema)
> var (topLevelColumnName, nestedColumn) = schema.columns.head
> ...
> {code}
>  # *The guard only checks the immediate base.* For {{{}COALESCE(r1, 
> r2).a.b{}}}, the outer access's base is the field access {{{}COALESCE(r1, 
> r2).a{}}}, not a {{{}RexCall{}}}, so the guard passes.
>  # *{{NestedProjectionUtil.build}} fails without saying so.* In 
> {{{}NestedSchemaExtractor.visitFieldAccess{}}}, {{internalVisit}} reaches the 
> {{RexCall}} ({{{}case expr => expr.accept(this){}}}). That registers the 
> call's operands ({{{}r1{}}}, {{{}r2{}}}) as top-level leaf columns, then 
> returns {{success = false}} and exits early. {{build()}} returns that schema 
> anyway.
>  # *The converter trusts the schema.* It takes {{schema.columns.head}} 
> ({{{}r1{}}}) and builds a {{{}NestedFieldReferenceExpression(["r1"]){}}}. It 
> gives it the type of the original field access ({{{}STRING{}}}), so the 
> corrupted predicate still passes type checking.
>  # {*}The corrupted expression becomes the remaining filter{*}, and {{r2}} is 
> pruned from the scan's columns.
> A single level of access ({{{}f(x).a{}}}) is caught by the existing guard; 
> two or more levels are not.
> h3. Proposed fix
> Follow the chain of field accesses down to its base, and don't convert unless 
> that base is a {{{}RexInputRef{}}}:
> {code:scala}
> @tailrec
> private def baseOf(e: RexNode): RexNode = e match {
>   case f: RexFieldAccess => baseOf(f.getReferenceExpr)
>   case other => other
> }
> override def visitFieldAccess(fieldAccess: RexFieldAccess): 
> Option[ResolvedExpression] = {
>   if (!baseOf(fieldAccess).isInstanceOf[RexInputRef]) {
>     return None
>   }
>   ...
> }
> {code}
> Optionally, {{NestedSchemaExtractor}} could also stop registering a call's 
> operands as leaf columns when called from this path, or the converter could 
> check {{{}build(){}}}'s result before using {{{}schema.columns.head{}}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to