vanzin commented on issue #24879: [SPARK-28042][K8S] Support using hostpath volume mount as local storage URL: https://github.com/apache/spark/pull/24879#issuecomment-511614793 A few things: - you explained this in comments. But the PR title explicitly talks about host paths and this is not only for host paths. That should be fixed. - I'm a little confused about how someone would write their configuration to trigger this. The second probably means better documentation, which you'll need to update. IIUC you list your local dirs in the configuration, and to use an existing volume you have to name it `spark-local-dir-$x` where `$x` is the index of the local dir in that array. With the current config-based volume mounts I don't think you can do much better. But it's kinda awkward, and I'd like to propose a different approach, which relies on pod templates. Basically, `LocalDirsFeatureStep` could detect all volumes (mounts?) with some Spark-specific label that indicates it should be used as temp storage, and update the configuration based on that. (e.g. all volume mounts with label `spark-local-dir`; I'm kinda assuming you can add labels to anything). If you're using the old config-based mounts, you wouldn't get this feature. But I think we should be moving to pod templates as a way to configure this stuff anyway. (And I hope we can with time remove all the non-pod-template configs to make things less confusing.) That also would not require changing the order of the feature steps (although there is no harm in doing that).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
