Yahya Kisana created SPARK-60410:
------------------------------------
Summary: Pushed filters on a whole struct are re-typed after
nested schema pruning in DSv2
Key: SPARK-60410
URL: https://issues.apache.org/jira/browse/SPARK-60410
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yahya Kisana
{{DataSourceV2ScanRelation.outputBoundPushedFilters}} rebinds the pushed
filters to the scan output with {{ProjectionOverSchema}} and keeps a remapped
filter when {{remapped.references.subsetOf(outputSet)}}. It does not check
whether the filter's value was narrowed. {{ProjectionOverSchema}} re-types a
whole struct attribute to its pruned type, so a filter on the whole struct is
re-typed after nested schema pruning.
Example: a source that fully pushes {{s = named_struct('x', 1L, 'y', 2L)}},
supports nested schema pruning and returns {{reflectsFullyPushedDownFilters()
== false}}. For {{SELECT s.y FROM t WHERE s = named_struct('x', 1L, 'y', 2L)}},
pruning ignores the fully pushed filter, so {{s}} narrows to {{struct<y>}}. The
filter then compares {{s: struct<y>}} with a {{struct<x, y>}} literal, and
{{withPostPushdownAdjustmentFilters}} adds it back as a {{Filter}}. That plan
is unresolved, so plan validation should fail the query. An {{IN}} list of more
than 10 values becomes an {{InSet}}, which has no input type check, so it would
compare {{y}} with the {{x}} of each literal and drop rows instead.
The inferred-filter remap in {{V2ScanRelationPushDown}} has the same gap.
Found by code reading during the review of SPARK-60011
(https://github.com/apache/spark/pull/59387). Not reproduced, and no in-tree
connector is known to hit it. SPARK-60011 drops a scan ordering whose value
pruning narrows ({{DataTypeUtils.sameType}} on the bound expression); a similar
check would fit here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
