Chen Zhang created FLINK-41524:
----------------------------------

             Summary: Filter pushdown rewrites a nested field access on a 
function result (f(x).a.b) into a wrong reference
                 Key: FLINK-41524
                 URL: https://issues.apache.org/jira/browse/FLINK-41524
             Project: Flink
          Issue Type: Bug
            Reporter: Chen Zhang


When a WHERE predicate accesses a field two or more levels deep on the result 
of a function call, e.g. {{{}COALESCE(r1, r2).a.b = 'x'{}}}, and the table 
source implements {{{}SupportsFilterPushDown{}}}, the planner corrupts the 
predicate. The function call and the field-access chain are dropped, and the 
predicate becomes {{{}r1 = 'x'{}}}. That predicate is:
 * passed to {{{}applyFilters(){}}},
 * kept as the remaining filter in the {{Calc}} above the scan,
 * and used by projection pushdown, which prunes the other arguments 
({{{}r2{}}}).

The query silently returns wrong results. Disabling 
{{table.optimizer.source.predicate-pushdown-enabled}} gives correct results.
h3. Steps to reproduce

Data file {{{}/tmp/repro/part-0.json{}}}:
{noformat}
{"r1":{"a":{"b":"x"}},"r2":{"a":{"b":"y"}},"s":"s1"}
{"r1":null,"r2":{"a":{"b":"x"}},"s":"s2"}
{noformat}
{code:sql}
CREATE TEMPORARY TABLE t (
  r1 ROW<a ROW<b STRING>>,
  r2 ROW<a ROW<b STRING>>,
  s  STRING
) WITH ('connector'='filesystem', 'path'='file:///tmp/repro', 'format'='json');

SELECT s FROM t WHERE COALESCE(r1, r2).a.b = 'x';
{code}
*Expected:* {{{}s1{}}}, {{s2}}
*Actual:* 0 rows

Plan with pushdown enabled (default):
{noformat}
Calc(select=[s], where=[(r1 = 'x')])
+- TableSourceScan(table=[[default_catalog, default_database, t, filter=[=(r1, 
_UTF-16LE'x')], project=[r1, s], metadata=[]]], fields=[r1, s])
{noformat}
Plan with {{{}SET 'table.optimizer.source.predicate-pushdown-enabled' = 
'false';{}}}:
{noformat}
Calc(select=[s], where=[(COALESCE(r1, r2).a.b = 'x')])
+- TableSourceScan(table=[[default_catalog, default_database, t]], fields=[r1, 
r2, s])
{noformat}
h3. Root cause

In {{RexNodeToExpressionConverter.visitFieldAccess}} 
({{{}RexNodeExtractor.scala{}}}):
{code:scala}
fieldAccess.getReferenceExpr match {
  // push down on nested field inside a composite type like map or array is not 
supported
  case _: RexCall => return None
  case _ =>
}
val schema = NestedProjectionUtil.build(Collections.singletonList(fieldAccess), 
dataType)
val fieldIndices = NestedProjectionUtil.convertToIndexArray(schema)
var (topLevelColumnName, nestedColumn) = schema.columns.head
...
{code}
 # *The guard only checks the immediate base.* For {{{}COALESCE(r1, 
r2).a.b{}}}, the outer access's base is the field access {{{}COALESCE(r1, 
r2).a{}}}, not a {{{}RexCall{}}}, so the guard passes.
 # *{{NestedProjectionUtil.build}} fails without saying so.* In 
{{{}NestedSchemaExtractor.visitFieldAccess{}}}, {{internalVisit}} reaches the 
{{RexCall}} ({{{}case expr => expr.accept(this){}}}). That registers the call's 
operands ({{{}r1{}}}, {{{}r2{}}}) as top-level leaf columns, then returns 
{{success = false}} and exits early. {{build()}} returns that schema anyway.
 # *The converter trusts the schema.* It takes {{schema.columns.head}} 
({{{}r1{}}}) and builds a {{{}NestedFieldReferenceExpression(["r1"]){}}}. It 
gives it the type of the original field access ({{{}STRING{}}}), so the 
corrupted predicate still passes type checking.
 # {*}The corrupted expression becomes the remaining filter{*}, and {{r2}} is 
pruned from the scan's columns.

A single level of access ({{{}f(x).a{}}}) is caught by the existing guard; two 
or more levels are not.
h3. Proposed fix

Follow the chain of field accesses down to its base, and don't convert unless 
that base is a {{{}RexInputRef{}}}:
{code:scala}
@tailrec
private def baseOf(e: RexNode): RexNode = e match {
  case f: RexFieldAccess => baseOf(f.getReferenceExpr)
  case other => other
}

override def visitFieldAccess(fieldAccess: RexFieldAccess): 
Option[ResolvedExpression] = {
  if (!baseOf(fieldAccess).isInstanceOf[RexInputRef]) {
    return None
  }
  ...
}
{code}
Optionally, {{NestedSchemaExtractor}} could also stop registering a call's 
operands as leaf columns when called from this path, or the converter could 
check {{{}build(){}}}'s result before using {{{}schema.columns.head{}}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to