Bogdan Raducanu created SPARK-26812:
---------------------------------------
Summary: PushProjectionThroughUnion nullability issue
Key: SPARK-26812
URL: https://issues.apache.org/jira/browse/SPARK-26812
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0
Reporter: Bogdan Raducanu
Union output data types are the output data types of the first child.
However the other union children may have different values nullability.
This means that we can't always push down a project on the children.
To reproduce
{code}
Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1")
sql("SELECT 1 AS b").write.saveAsTable("table2")
sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL
SELECT a FROM table1")
sql("select * from test1").show
{code}
This fails becaus the plan is no longer resolved.
The plan is broken by the PushProjectionThroughUnion rule which pushed down a
cast to map<string,string> with values nullability=true on a child with type
map<string, string> with values nullability=false.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
