[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406612146
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
In the above, `options.readOnly=None` is added because we don't failure at
missing `options.readOnly`.
If possible, it would be great if we can use `mount.readOnly` only.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406612146
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
In the above, `options.readOnly=None` is added because we don't failure at
missing `options.readOnly`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=true, options.readOnly=None
4. mount.readOnly=false, options.readOnly=true
5. mount.readOnly=false, options.readOnly=false
6. mount.readOnly=false, options.readOnly=None
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=false, options.readOnly=true
4. mount.readOnly=false, options.readOnly=false
5. mount.readOnly=false, options.readOnly=None
6. mount.readOnly=false, options.readOnly=None
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611618
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
Review comment:
In this case, we don't need
`sparkConf.set("test.nfs.volumeName.options.readOnly", "true")`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=false, options.readOnly=true
4. mount.readOnly=false, options.readOnly=false
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611075
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === true)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", None, "nfs.example.com"))
+ }
+
+ test("Parses read/write nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === false)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", Some(false), "nfs.example.com"))
+ }
+
+ test("Fails on missing path option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.pth", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val e = intercept[NoSuchElementException] {
+ KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf, "test.")
+}
+assert(e.getMessage.contains("nfs.volumeName.options.path"))
+ }
+
+ test("Fails on missing server option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.s", "nfs.example.com")
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611010
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === true)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", None, "nfs.example.com"))
+ }
+
+ test("Parses read/write nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === false)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", Some(false), "nfs.example.com"))
+ }
+
+ test("Fails on missing path option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.pth", "/share")
Review comment:
Shall we remove this line instead of using `pth`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406608506
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
Review comment:
nit.
```
- test("Parses nfs volumes correctly") {
+ test("Parses read-only nfs volumes correctly") {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
