Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19464
Interesting. On the one hand I don't like adding yet another flag that
changes behavior, when the user often can't meaningfully decide to set it.
There is probably no value in processing an empty partition, sure. Then again
it does change behavior slightly, and I wonder if that impacts assumptions that
apps rely on somehow.
If there's no reason to expect downside, we could do this in Spark 3.x, or
make the change now but yes introduce a flag as a safety valve to go back to
old behavior, leaving the default to true.
But first are there any known impacts to skipping the empty partitions?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
