Chen Zhang created FLINK-41524:
----------------------------------
Summary: Filter pushdown rewrites a nested field access on a
function result (f(x).a.b) into a wrong reference
Key: FLINK-41524
URL: https://issues.apache.org/jira/browse/FLINK-41524
Project: Flink
Issue Type: Bug
Reporter: Chen Zhang
When a WHERE predicate accesses a field two or more levels deep on the result
of a function call, e.g. {{{}COALESCE(r1, r2).a.b = 'x'{}}}, and the table
source implements {{{}SupportsFilterPushDown{}}}, the planner corrupts the
predicate. The function call and the field-access chain are dropped, and the
predicate becomes {{{}r1 = 'x'{}}}. That predicate is:
* passed to {{{}applyFilters(){}}},
* kept as the remaining filter in the {{Calc}} above the scan,
* and used by projection pushdown, which prunes the other arguments
({{{}r2{}}}).
The query silently returns wrong results. Disabling
{{table.optimizer.source.predicate-pushdown-enabled}} gives correct results.
h3. Steps to reproduce
Data file {{{}/tmp/repro/part-0.json{}}}:
{noformat}
{"r1":{"a":{"b":"x"}},"r2":{"a":{"b":"y"}},"s":"s1"}
{"r1":null,"r2":{"a":{"b":"x"}},"s":"s2"}
{noformat}
{code:sql}
CREATE TEMPORARY TABLE t (
r1 ROW<a ROW<b STRING>>,
r2 ROW<a ROW<b STRING>>,
s STRING
) WITH ('connector'='filesystem', 'path'='file:///tmp/repro', 'format'='json');
SELECT s FROM t WHERE COALESCE(r1, r2).a.b = 'x';
{code}
*Expected:* {{{}s1{}}}, {{s2}}
*Actual:* 0 rows
Plan with pushdown enabled (default):
{noformat}
Calc(select=[s], where=[(r1 = 'x')])
+- TableSourceScan(table=[[default_catalog, default_database, t, filter=[=(r1,
_UTF-16LE'x')], project=[r1, s], metadata=[]]], fields=[r1, s])
{noformat}
Plan with {{{}SET 'table.optimizer.source.predicate-pushdown-enabled' =
'false';{}}}:
{noformat}
Calc(select=[s], where=[(COALESCE(r1, r2).a.b = 'x')])
+- TableSourceScan(table=[[default_catalog, default_database, t]], fields=[r1,
r2, s])
{noformat}
h3. Root cause
In {{RexNodeToExpressionConverter.visitFieldAccess}}
({{{}RexNodeExtractor.scala{}}}):
{code:scala}
fieldAccess.getReferenceExpr match {
// push down on nested field inside a composite type like map or array is not
supported
case _: RexCall => return None
case _ =>
}
val schema = NestedProjectionUtil.build(Collections.singletonList(fieldAccess),
dataType)
val fieldIndices = NestedProjectionUtil.convertToIndexArray(schema)
var (topLevelColumnName, nestedColumn) = schema.columns.head
...
{code}
# *The guard only checks the immediate base.* For {{{}COALESCE(r1,
r2).a.b{}}}, the outer access's base is the field access {{{}COALESCE(r1,
r2).a{}}}, not a {{{}RexCall{}}}, so the guard passes.
# *{{NestedProjectionUtil.build}} fails without saying so.* In
{{{}NestedSchemaExtractor.visitFieldAccess{}}}, {{internalVisit}} reaches the
{{RexCall}} ({{{}case expr => expr.accept(this){}}}). That registers the call's
operands ({{{}r1{}}}, {{{}r2{}}}) as top-level leaf columns, then returns
{{success = false}} and exits early. {{build()}} returns that schema anyway.
# *The converter trusts the schema.* It takes {{schema.columns.head}}
({{{}r1{}}}) and builds a {{{}NestedFieldReferenceExpression(["r1"]){}}}. It
gives it the type of the original field access ({{{}STRING{}}}), so the
corrupted predicate still passes type checking.
# {*}The corrupted expression becomes the remaining filter{*}, and {{r2}} is
pruned from the scan's columns.
A single level of access ({{{}f(x).a{}}}) is caught by the existing guard; two
or more levels are not.
h3. Proposed fix
Follow the chain of field accesses down to its base, and don't convert unless
that base is a {{{}RexInputRef{}}}:
{code:scala}
@tailrec
private def baseOf(e: RexNode): RexNode = e match {
case f: RexFieldAccess => baseOf(f.getReferenceExpr)
case other => other
}
override def visitFieldAccess(fieldAccess: RexFieldAccess):
Option[ResolvedExpression] = {
if (!baseOf(fieldAccess).isInstanceOf[RexInputRef]) {
return None
}
...
}
{code}
Optionally, {{NestedSchemaExtractor}} could also stop registering a call's
operands as leaf columns when called from this path, or the converter could
check {{{}build(){}}}'s result before using {{{}schema.columns.head{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)