Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/20091
Thanks for coding it up @jiangxb1987 !
So if I understand it correctly, the requirements where the PR helps with
are :
* Max partitioner is not eligible since it is atleast an order smaller, and
* User has explicitly set 'spark.default.parallelism', and
* Value of 'spark.default.parallelism' is lower than max partitioner
** Since max partitioner was discarded due to being atleast an order
smaller, default parallelism is worse - even though user specified.
Does it impact any other usecase or flow ? I want to make sure I am not
missing anything.
If strictly this, then I agree that the PR makes sense. It is a fairly
suboptimal situation which we are hopefully not worsening - even if we are
ignoring user specified value (by relying on existing behavior :-) )
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
