ulysses-you commented on code in PR #47418:
URL: https://github.com/apache/spark/pull/47418#discussion_r1716185322
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:
##########
@@ -194,6 +197,22 @@ object DataSourceAnalysis extends Rule[LogicalPlan] {
targetPartitionSchema = t.partitionSchema)
Project(projectList, query)
} else {
+ query.transformWithPruning(!_.containsAnyPattern(UNION, JOIN)) {
Review Comment:
I do not know how to review it, there are many issues to me with a read
roughly ...
- It is not the right way to get the filter adjacent to scan, see
`PhysicalOperation`
- what happens if there is a query like `insert into table t1 partition(p)
select (p+1) as p from t where p = 1`
- what happens if there is a query like `insert into table t1 partition(p)
select p from t where p = 1 and p = 2`
- what happens if the filter is on the second partition column `insert into
table t1 partition(p1, p2) select * from t2 where p2=1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
