warrenzhu25 commented on PR #41076: URL: https://github.com/apache/spark/pull/41076#issuecomment-1537293952
> Thank you for pinging me. Could you improve this proposal more, @warrenzhu25 ? > > * This PR claims that data migration can hurt performance, but the code is applied in all cases including the case data migration (shuffle/rdd) is disabled. Moreover, in general, when the migration data size is small, the claim is invalid. > * In the same way, I can imagine this PR introduces another regression in terms of resource utilizations. For example, Spark Thrift Server can decommission all executors at the same time, but this configuration (< 0.1) may hurt the speed of scaling down. > * It would be great if we can have a clear documentation about the relationship between this and other decommission configs. `maxRatio` has default value 1, so it's same as current behavior. Users has full control of this ratio based on their judgement of storage migration size and impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
