shaneknapp commented on issue #23514: [SPARK-24902][K8s] Add PV integration tests URL: https://github.com/apache/spark/pull/23514#issuecomment-476368980 ok... more testing completed, and more data about the failure(s). 1) minikube mount is instantiated as so: `minikube mount /home/jenkins/src/spark/tmp:/tmp/spark-k8s-integration-tests --9p-version=9p2000.L --uid=1001 --gid=1001 &` 2) the PV tests *cannot* create files or subdirs in the mounted directory from the container (verified by sshing in to the container during test runtime) 3) if i make the mounted directory globally RWX on the host machine, i am able to create files in the root of the mounted dir, as well as one level of subdirs. however, i am unable to create any subdirs below those child dirs (see output of this below). so yeah, this is almost definitely a bug in the kvm2 VM driver. next steps? i guess i'll talk to my minikube friend and see if there's any plan to fix this (and/or see if there's an existing issue for this). another option is to move off of kvm2 to another VM driver, but this is a seriously non-trivial amount of work and i'd much rather avoid doing this. Mkdirs error from the test(s) -- this matches the behavior i saw when logging in manually and trying to create my own dirs: ``` 19/03/25 20:13:05 WARN TaskSetManager: Lost task 1.3 in stage 0.0 (TID 7, 172.17.0.5, executor 1): java.io.IOException: Mkdirs failed to create file:/opt/spark/pv-tests/dfs_read_write_test/_temporary/0/_temporary/attempt_20190325201302_0001_m_000001_3 (exists=false, cwd=file:/opt/spark/work-dir) ``` output from logging in to the container directly: ``` kubectl -n ed0ff1e340b849c1be3fedec2eabaa74 exec -it spark-test-app-5c548894c0ae4e13af4f97fc279d52c6 -- /bin/bash bash-4.4$ cd /opt/spark/pv-tests bash-4.4$ ls tmp4781285275433042132.txt bash-4.4$ touch ASDF touch: ASDF: Permission denied bash-4.4$ mkdir test mkdir: can't create directory 'test': Permission denied bash-4.4$ ls -ld . drwxrwxr-x 1 1001 1001 4096 Mar 25 20:12 . bash-4.4$ id uid=185(185) gid=0(root) <SHANE RUNS chmod -R 0777 ~/src/spark/tmp/ ON THE HOST SERVER> bash-4.4$ ls -ld . drwxrwxrwx 1 1001 1001 4096 Mar 25 20:12 . bash-4.4$ touch ASDF bash-4.4$ ls ASDF tmp4781285275433042132.txt bash-4.4$ mkdir TEST bash-4.4$ ls ASDF TEST tmp4781285275433042132.txt bash-4.4$ cd TEST bash-4.4$ ls bash-4.4$ pwd /opt/spark/pv-tests/TEST bash-4.4$ mkdir TESTSUBDIR mkdir: can't create directory 'TESTSUBDIR': Permission denied bash-4.4$ ls -ld . drwxr-xr-x 1 1001 1001 4096 Mar 25 20:13 . bash-4.4$ cd .. bash-4.4$ ls -ld drwxrwxrwx 1 1001 1001 4096 Mar 25 20:13 . bash-4.4$ ls ASDF TEST tmp4781285275433042132.txt bash-4.4$ ls -l total 5 -rw-r--r-- 1 1001 1001 0 Mar 25 20:13 ASDF drwxr-xr-x 1 1001 1001 4096 Mar 25 20:13 TEST -rwxrwxrwx 1 1001 1001 8 Mar 25 20:12 tmp4781285275433042132.txt bash-4.4$ pwd /opt/spark/pv-tests bash-4.4$ exit ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
