pralabhkumar commented on PR #37203: URL: https://github.com/apache/spark/pull/37203#issuecomment-1189298711
> > IMHO , this should be randomized , so that all the directories have equal changes of pushing the data as was done on yarn side > > Was there an actual problem that occurred because these were not random? @tgravescs Yes we are seeing same problem(https://issues.apache.org/jira/browse/SPARK-24992) in our K8s cluster, where most of the time one disk get filled. > > how was this tested? I had tested this via unit test cases . Also ran internally on K8s cluster and seen executor logs . There have seen the disk get randomized on different executors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
