Github user davies commented on the issue:
https://github.com/apache/spark/pull/14452
@viirya Had a quick look on this one, It seems that we are trying to make
the logical plans looks better, but does not help much for actual physical plan
or runtime. For example, for self join, the plan had to executed twice (because
of the current poll model) without materialization. If they have Shuffle or
Broadcast, we can already re-use them. The only benefit I can imagine is that
this could help to prevent push different filter on both side which prevent
them could be reused (the result will be different). So the question is that
should we push down this filter or not? It's not obvious to me, it could be
good or bad based on use cases.
So I don't see much benefits from this PR (not a trivial change). Maybe I
misunderstand something, please correct me by show some real use cases, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
