Holden Karau created SPARK-44951: ------------------------------------ Summary: Improve Spark Dynamic Allocation Key: SPARK-44951 URL: https://issues.apache.org/jira/browse/SPARK-44951 Project: Spark Issue Type: Improvement Components: Kubernetes, Spark Core, YARN Affects Versions: 4.0.0 Reporter: Holden Karau
For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential ideas here includes the following: * Plug-gable DEA algorithms * How to reduce wastage on the RM side? Sometimes the driver asks for some units of resources. But when RM provisions them, the driver cancels it. * Support for "warm" executor pools which are not tied to a particular driver but start and wait for a driver to connect to them to "claim" them. * More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow the developer to choose between cost and runtime. Sometimes developers might be ok to pay higher costs for faster execution. * Use previous run information to inform future runs * Better selection of executors to be scaled down -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org