shaneknapp edited a comment on issue #23514: [SPARK-24902][K8s] Add PV integration tests URL: https://github.com/apache/spark/pull/23514#issuecomment-476771619 > @shaneknapp things run smoothly on ubuntu with kvm2 what is the bug you are referring to? > I pasted the runs above. Try use `--gid=0` so you can create the files (I didnt use --uid=1001 --gid=1001 check https://unix.stackexchange.com/questions/44077/what-does-it-mean-to-be-in-group-0), pls follow what I did above. sorry, i missed those args in the earlier reply... so i hacked my testing env and guess what? ``` Run starting. Expected test count is: 16 KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - Test PVs with local storage Run completed in 8 minutes, 34 seconds. Total number of tests run: 16 Suites: completed 2, aborted 0 Tests: succeeded 16, failed 0, canceled 0, ignored 0, pending 0 All tests passed. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 3.632 s] [INFO] Spark Project Tags ................................. SUCCESS [ 3.029 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 2.364 s] [INFO] Spark Project Networking ........................... SUCCESS [ 3.811 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 2.355 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 2.558 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 3.139 s] [INFO] Spark Project Core ................................. SUCCESS [ 20.909 s] [INFO] Spark Project Kubernetes Integration Tests ......... SUCCESS [09:21 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10:04 min [INFO] Finished at: 2019-03-26T10:41:23-07:00 [INFO] ------------------------------------------------------------------------ ``` great success! :) now, before i modify the k8s build script and this is committed, there are a couple of things we will need to tweak: in `./resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala`, line 185: `HOST_PATH` is currently incorrect for running on the prod jenkins workers. the correct path is passed in as an environment variable named `$WORKSPACE` when the build is launched. can you update `PVTestsSuite.scala` to set `HOST_PATH` to be `$WORKSPACE/tmp`? once this is pushed, i'll update the build to create/destroy the mount and retrigger the builds. i *REALLY* do not like the fact that we will need a separate `minikube mount...` command in the build script, but hopefully we'll be able to pass in the necessary `minikube mount` args via `minikube start` in the not-too-distant future. we're super close!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
