Github user mengxr commented on the issue:
https://github.com/apache/spark/pull/21927
@jiangxb1987 Second thought: `PartitionPruningRDD` is just an
implementation of RDD. Every user / developer can implement a similar one. Also
this doesn't handle the case mentioned by @felixcheung :
`a.union(b).barrier()`. So I'm thinking about checking number of partitions
instead of instances of `PartitionPruningRDD` in this PR. Basically, we check
the input RDD and all its parents have the same number of partitions. If not,
we throw an error message like "Barrier execution mode doesn't support
partition union / pruning.". Thoughts?
cc: @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
