warrenzhu25 commented on PR #40730: URL: https://github.com/apache/spark/pull/40730#issuecomment-1502631790
> I understand the intention but there is a chance of instability due to `OutOfDisk` and sometimes `OutOfMemory`. In addition, bin-packed executors could work slower due to the network traffic congestion. Do you have some production results about the real benefits, @warrenzhu25 ? In prod, we have seen 20~50 percent resource save based on executor size, the more saving with more cpu cores per executor. For the issue could be caused by bin-packed executors, it's still possible to happen when all executor cores are occupied. If it indeed happen, the better solution might be large partition num. The purpose of change is to provide one more scheduling option, customer still can use the old way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
