viirya commented on PR #55642: URL: https://github.com/apache/spark/pull/55642#issuecomment-4358068276
> `ExecutorPVCResizePlugin` is supposed to be used for homogeneous PVC shuffle storage case (the following) and not designed for those heterogeneous scenario. Thanks, that helps clarify the intended scope. If the intended contract is a homogeneous PVC shuffle/local-dir storage setup, then I think we should make that contract explicit. Right now the implementation resizes every PVC mounted by the executor container, but the docs/configs only say “executor PVC” and do not warn that heterogeneous PVC mounts are unsupported. Users may reasonably combine Spark local-dir PVCs with other executor PVC mounts via pod templates, and those unrelated PVCs would also be resized. Could we either document this limitation very explicitly, or preferably restrict the implementation to Spark local-dir PVC mounts only? That would preserve the simple homogeneous design while avoiding accidental resize of unrelated PVCs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
