serdar onur created SPARK-31649:
-----------------------------------
Summary: Spread partitions evenly to spark executors
Key: SPARK-31649
URL: https://issues.apache.org/jira/browse/SPARK-31649
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 2.4.4
Reporter: serdar onur
The year is 2020 and I am still trying to find a solution to this. I totally
understand what [~thunderstumpges] was trying to achieve and I am trying to
achieve the same. For a tool like spark, it is unacceptable not to be able to
distribute the created partitions to the executors evenly. You know, we can
create a custom partitioner to distribute the data to the partitions evenly by
creating our own partition index. I was under the impression that a similar
approach could be applied to spread these partitions to the executors
evenly(using some sort of executor index for selection of executors during
partition distribution). I have been googling this for a day now and I am very
disappointed to say that up to now this seems to be not possible.
Note: I am disappointed that the issue below was put into resolved state
without actually doing anything about it.
https://issues.apache.org/jira/browse/SPARK-19371
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
