Jungtaek Lim created SPARK-47305:
------------------------------------
Summary: PruneFilters incorrectly tags isStreaming flag when
replacing child of Filter with LocalRelation
Key: SPARK-47305
URL: https://issues.apache.org/jira/browse/SPARK-47305
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.5.1, 3.5.0, 3.4.1, 3.4.0, 3.4.2, 4.0.0
Reporter: Jungtaek Lim
This seems to be a very old bug in optimizer. Related ticket:
https://issues.apache.org/jira/browse/SPARK-21765
When filter is evaluated to be always false, PruneFilters replaces the filter
with empty LocalRelation, which effectively prunes filter. The logic cares
about migration of the isStreaming flag, but incorrectly migrated in some case,
via picking up the value of isStreaming flag from root node rather than filter
(or child).
isStreaming flag is true if the value of isStreaming flag from any of children
is true. Flipping the coin, some children might have isStreaming flag as
"false". If the filter being pruned is a descendant to such children (in other
word, ancestor of streaming node), LocalRelation is incorrectly tagged as
streaming where it should be batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
