Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/16005
@nsyca +1000 on whitelisting operators. That is what we should have done
from the start.
Let's break it down:
1. `LeafNode`s should not be a problem. We don't need to explicitly handle
them
2. We should allow the following `UnaryNode`: `Project`, `Filter`,
`Aggregate`, `SubqueryAlias`, `Distinct`, `Generate` (only when `join=true`),
`BroadcastHint`, `Sort`, `Repartition` & `RedistributeData` (parent of
`SortPartitions` and `RepartitionByExpression`). We need to find out what other
systems allow for `Window`.
3. The only `BinaryNode` we should allow is `Join` with special cases for
`Left`/`Right`/`Full`. We should also make sure that the `LeftAnti` and
`LeftSemi` are handled properly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
