wuyi created SPARK-48636:
----------------------------
Summary: Event driven block manager decommissioner
Key: SPARK-48636
URL: https://issues.apache.org/jira/browse/SPARK-48636
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.3.4, 3.5.1, 3.2.4, 3.1.3, 4.0.0
Reporter: wuyi
The current blockmanager decommisssioner uses the periodic threads to refresh
blocks/ peers, monitor progress. It can be low-effeicent this way. For example,
in the worst case, it takes 30s (by default) at most for an executor to exit
itself even if all the blocks have been migrated. The cause is that the
migration status is checked every 30s (by default).
So this ticket proposes the blockmanager decommissioner to leverage the events
driven way to improve its effeciency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
