onursatici commented on issue #27695: URL: https://github.com/apache/spark/pull/27695#issuecomment-616621311
Hey @dongjoon-hyun , sorry for the late response. You are right that the feature I wan't with this PR is to mainly increase parallelism while keeping the cpu resource requests low in K8S. `spark.driver.cores` controls the number of threads in various components, including some network components that are mainly blocked by I/O. The ability to increase this while keeping the cpu requests the same would allow users to better utilise their K8S clusters, especially if their workloads are I/O bound ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
