shaneknapp edited a comment on issue #23514: [SPARK-24902][K8s] Add PV integration tests URL: https://github.com/apache/spark/pull/23514#issuecomment-462578138 i'm really running out of steam on trying to get this test to work. :\ before you read the rest of my response below, i would strongly suggest you set up a minikube instance w/a kvm2 VM backend and see if you can get this stuff working yourself. :) i have a strong feeling that you're not correctly creating the mounts for the pod. and after spending another afternoon banging on this, i feel that i finally figured out what's going on. TL;DR: **this will work, but you're just missing an intermediate step (the mount on the VM).** LONGER VERSION: paths and mounts. let's discuss... i think there might be something wrong in how you're approaching this. things are mounted in the following order: 1) `<path on local machine>` is mounted as... 2) `<path on KVM hypervisor>`, which is (eventually) mounted as... 3) `<path in the pod itself>`. (1) and (2) are defined when i launch minikube w/the `--mount-string` and `--mount` options. eg: ``` minikube --vm-driver=kvm2 start --memory 6000 --cpus 8 --mount-string="/jenkins/src/spark/tmp:/tmp/jenkins-k8s-testing" --mount ``` (3) is defined in `PVTestsSuite.scala` as `val CONTAINER_MOUNT_PATH = "/opt/spark/pv-tests"` (1) seems to need to be the same as `val HOST_PATH = "<somedir>"`, otherwise `createTempFile()` won't work (as it's not being run in the testing pod). however, when you're creating the PV and linking it to the volume, you're also using `HOST_PATH`... but shouldn't it be mount as seen on the VM layer (eg: mount 2), and not the bare metal FS? if i create my own pod and connect to the cluster that the test uses, i can actually get things to work! running this command to fire up the cluster: ``` $ minikube --vm-driver=kvm2 start --memory 6000 --cpus 8 --mount-string="/home/jenkins/src/spark/tmp:/tmp/jenkins-k8s-testing" --mount Starting local Kubernetes v1.10.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... Setting up hostmount on /home/jenkins/src/spark/tmp:/tmp/testing-k8s... Kubectl is now configured to use the cluster. Loading cached images from config file. ``` then ssh in and see what if my mount is there: ``` $ minikube ssh <silly motd deleted> $ mount | grep "testing-k8s" 192.168.39.1 on /tmp/testing-k8s type 9p (rw,relatime,sync,dirsync,trans=tcp,port=41201,dfltuid=1001,dfltgid=1001,version=9p2000.u,msize=262144) ``` if i go to that dir (`/tmp/testing-k8s `) and touch a file, it shows up on my local FS. as expected. now, if i create a pod and set up the `volumes` and `volumeMounts` like this: ``` $ cat k8s-container-login-pv-testing.sh #!/bin/bash kubectl run -i --rm --tty ubuntu --overrides=' { "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "ubuntu-testing" }, "spec": { "containers": [ { "name": "ubuntu", "image": "ubuntu:16.04", "args": [ "bash" ], "stdin": true, "stdinOnce": true, "tty": true, "volumeMounts": [{ "mountPath": "/tmp", "name": "host-mount" }] } ], "volumes": [ { "name": "host-mount", "hostPath": { "path": "/tmp/testing-k8s" } } ] } } ' --image=ubuntu:16.04 --restart=Never -- bash ``` by logging in to that pod, and cd-ing to `/tmp`, i can touch files and see them show up on the local FS in `/home/jenkins/src/spark/tmp`. and finally, just because this reply isn't quite long enough, here's the PV, PVC and storageclass info: ``` $ kubectl describe pv Name: test-local-pv Labels: <none> Annotations: pv.kubernetes.io/bound-by-controller=yes StorageClass: test-local-storage Status: Bound Claim: a401685ff7d649a591f7622f4dc1105b/test-local-pvc Reclaim Policy: Retain Access Modes: RWO Capacity: 1Gi Message: Source: Type: LocalVolume (a persistent volume backed by local storage on a node) Path: /tmp/testing-k8s Events: <none> $ kubectl describe pvc --all-namespaces Name: test-local-pvc Namespace: a401685ff7d649a591f7622f4dc1105b StorageClass: test-local-storage Status: Bound Volume: test-local-pv Labels: <none> Annotations: pv.kubernetes.io/bind-completed=yes pv.kubernetes.io/bound-by-controller=yes Capacity: 1Gi Access Modes: RWO Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 24m (x2 over 24m) persistentvolume-controller waiting for first consumer to be created before binding $ kubectl -n a401685ff7d649a591f7622f4dc1105b describe storageclass test-local-storage Name: test-local-storage IsDefaultClass: No Annotations: <none> Provisioner: kubernetes.io/no-provisioner Parameters: <none> Events: <none> ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
