Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19424
@rdblue we assume the previous batches should push down operators as close
to data source relations as possible. One special case is column pruning. The
`ColumnPruning` rule tries to generate a new `Project` node under every node
that has more input columns than it references, which means it will also
generate a `Project` under `Filter`, and this conflicts with
`PushDownPredicate` rule. Currently the conflict is resolved by a hacky way so
there are cases `Project` under `Filter`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
