[GitHub] [spark] tenglei commented on pull request #38181: [SPARK-40720][CORE] Fix spark-ui jobs status not updating under high concurrency scenario

Sun, 09 Oct 2022 06:09:41 -0700 


tenglei commented on PR #38181:
URL: https://github.com/apache/spark/pull/38181#issuecomment-1272539284

   Set spark.scheduler.listenerbus.eventqueue.capacity = 500, and submit as 
many jobs as possible in 30 minutes
   
   Before this pr
   
   We can see there are 806(completed)+66(active) of jobs in spark-ui
   
![image](https://user-images.githubusercontent.com/13283099/194758325-e89ee1c8-9676-429f-828a-43460eb6525f.png)
   
![image](https://user-images.githubusercontent.com/13283099/194758331-110a091c-e6e4-4b2c-b49a-dfba63a06c4a.png)
   
   Wait for 3h, there still lot of jobs in spark-ui that status not update
   
![image](https://user-images.githubusercontent.com/13283099/194758336-9d5f22f7-cb5a-4ca4-9c75-8b3472c92989.png)
   
   After this pr, and set 
   
   We can see there are 857(completed)+46(active) of jobs in spark-ui
   
![image](https://user-images.githubusercontent.com/13283099/194758503-81cd5ada-f22c-434e-9dec-774b2b4c6db5.png)
   
   Wait for 7 min, there 935(completed) of jobs and no active job
   
![image](https://user-images.githubusercontent.com/13283099/194758578-235e7c06-7e89-4814-9d2f-688a6f54325b.png)
   
   
   
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to