[ https://issues.apache.org/jira/browse/SPARK-47001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Holden Karau updated SPARK-47001: --------------------------------- Description: When pushing a filter down in a union the data type may not match exactly if the filter was constructed using the child dataframe reference. This is because the unions output is updated with a structype merge of union which can turn non-nullable to nullable. These are still the same column despite the different nullability so the filter should be safe to push down. As it currently stands we get an exception. (was: Right now it asserts exact equality but uses semanticEquality for candidacy, this can result in an unexpected exception in Optimizer.scala when pushing down semantically equal but different values.) Summary: Pushdown Verification in Optimizer.scala should support changed data types (was: Pushdown Verification in Optimizer.scala should use semantic equals) > Pushdown Verification in Optimizer.scala should support changed data types > -------------------------------------------------------------------------- > > Key: SPARK-47001 > URL: https://issues.apache.org/jira/browse/SPARK-47001 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.3.0 > Reporter: Holden Karau > Assignee: Holden Karau > Priority: Major > > When pushing a filter down in a union the data type may not match exactly if > the filter was constructed using the child dataframe reference. This is > because the unions output is updated with a structype merge of union which > can turn non-nullable to nullable. These are still the same column despite > the different nullability so the filter should be safe to push down. As it > currently stands we get an exception. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org