[GitHub] [spark] bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.

[GitBox]

bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time 
measure resource under utilization due to delay scheduling.
URL: https://github.com/apache/spark/pull/27207#issuecomment-611386010
 
 
   First manual test looks good. Ran a simple app which creates 1000 partitions 
all preferring executor 0.
   Each task sleeps 1 second. [Here is the app's 
code](https://github.com/bmarcott/spark-test-apps/blob/master/src/main/scala/TestLocalityWait.scala)
 . 
   New code utilized all executors, whereas with legacy flag enabled, only 
process local tasks were run, making it much slower.
   
   **Run with new command:**
   `../../projects/apache/spark/bin/spark-submit  --class "TestLocalityWait" 
--master spark://localhost:7077 --conf spark.executor.instances=4 --conf 
spark.executor.cores=2  target/scala-2.12/simple-project_2.12-1.0.jar`
   
   **Processed 326 partitions in 35 seconds:**
   <img width="1043" alt="image" 
src="https://user-images.githubusercontent.com/481161/78868871-7d582380-79f8-11ea-93d1-425bcdc5e856.png";>
   
   **Many tasks run at ANY locality level:**
   <img width="511" alt="image" 
src="https://user-images.githubusercontent.com/481161/78868894-8812b880-79f8-11ea-83bc-a620080ffe94.png";>
   
   
   ---------------------------------------------------------------------
   
   **Run with legacy flag set to true:**
   `../../projects/apache/spark/bin/spark-submit  --class "TestLocalityWait" 
--master spark://localhost:7077 --conf spark.executor.instances=4 --conf 
spark.executor.cores=2 --conf spark.locality.wait.legacyResetOnTaskLaunch=true 
target/scala-2.12/simple-project_2.12-1.0.jar`
   
   **Processed 146 partitions in 1.3 min:**
   <img width="1041" alt="image" 
src="https://user-images.githubusercontent.com/481161/78868626-1c305000-79f8-11ea-9386-1ea06e52870c.png";>
   
   **All tasks run at PROCESS_LOCAL locality level**
   <img width="523" alt="image" 
src="https://user-images.githubusercontent.com/481161/78868707-3c600f00-79f8-11ea-89bb-4dc08bbb4885.png";>
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org