tenglei opened a new pull request, #38181: URL: https://github.com/apache/spark/pull/38181
[SPARK-40720][CORE] Fix spark-ui jobs status not updating under high concurrency scenario ### What changes were proposed in this pull request? This PR proposes to optimizes the logic of the AsyncEventQueue queue to discard spark events, by adding a timeout configuration to preserve spark events as much as possible. ### Why are the changes needed? Spark-ui jobs status not updating under high concurrency scenario, this can cause confusion for some users ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? After setting spark.scheduler.listenerbus.eventqueue.timeout (e.g. 30s) set spark.scheduler.listenerbus.eventqueue.capacity as small as possible(e.g. 500) and submit as many jobs as possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
