This is the problem: > :/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar;/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
Seems like some code is confusing things when mixing OSes. It's using the Windows separator when building a command line ti be run on a Linux host. On Tue, Apr 10, 2018 at 11:02 AM, Dmitry <frostb...@gmail.com> wrote: > Previous example was bad paste( I tried a lot of variants, so sorry for > wrong paste ) > PS C:\WINDOWS\system32> spark-submit --master k8s://https://ip:8443 > --deploy-mode cluster --name spark-pi --class > org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 > --executor-memory 1G --conf spark.kubernete > s.container.image=andrusha/spark-k8s:2.3.0-hadoop2.7 > local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar > Returns > Image: > andrusha/spark-k8s:2.3.0-hadoop2.7 > Environment variables: > SPARK_DRIVER_MEMORY: 1g > SPARK_DRIVER_CLASS: org.apache.spark.examples.SparkPi > SPARK_DRIVER_ARGS: > SPARK_DRIVER_BIND_ADDRESS: > SPARK_MOUNTED_CLASSPATH: > /opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar;/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar > SPARK_JAVA_OPT_0: > -Dspark.kubernetes.driver.pod.name=spark-pi-46f48a0974d43341886076bc3c5f31c4-driver > SPARK_JAVA_OPT_1: > -Dspark.kubernetes.executor.podNamePrefix=spark-pi-46f48a0974d43341886076bc3c5f31c4 > SPARK_JAVA_OPT_2: -Dspark.app.name=spark-pi > SPARK_JAVA_OPT_3: > -Dspark.driver.host=spark-pi-46f48a0974d43341886076bc3c5f31c4-driver-svc.default.svc > SPARK_JAVA_OPT_4: -Dspark.submit.deployMode=cluster > SPARK_JAVA_OPT_5: -Dspark.driver.blockManager.port=7079 > SPARK_JAVA_OPT_6: -Dspark.master=k8s://https://ip:8443 > SPARK_JAVA_OPT_7: > -Dspark.jars=/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar,/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar > SPARK_JAVA_OPT_8: > -Dspark.kubernetes.container.image=andrusha/spark-k8s:2.3.0-hadoop2.7 > SPARK_JAVA_OPT_9: -Dspark.executor.instances=1 > SPARK_JAVA_OPT_10: -Dspark.app.id=spark-16eb67d8953e418aba96c2d12deecd11 > SPARK_JAVA_OPT_11: -Dspark.executor.memory=1G > SPARK_JAVA_OPT_12: -Dspark.driver.port=7078 > > > -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS > $SPARK_DRIVER_ARGS) > + exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java > -Dspark.app.id=spark-16eb67d8953e418aba96c2d12deecd11 > -Dspark.executor.memory=1G -Dspark.driver.port=7078 > -Dspark.driver.blockManager.port=7079 -Dspark.submit.deployMode=cluster > -Dspark.jars=/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar,/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar > -Dspark.master=k8s://https://172.20.10.12:8443 > -Dspark.kubernetes.executor.podNamePrefix=spark-pi-46f48a0974d43341886076bc3c5f31c4 > -Dspark.kubernetes.driver.pod.name=spark-pi-46f48a0974d43341886076bc3c5f31c4-driver > -Dspark.driver.host=spark-pi-46f48a0974d43341886076bc3c5f31c4-driver-svc.default.svc > -Dspark.app.name=spark-pi -Dspark.executor.instances=1 > -Dspark.kubernetes.container.image=andrusha/spark-k8s:2.3.0-hadoop2.7 -cp > ':/opt/spark/jars/*:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar;/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar' > -Xms1g -Xmx1g -Dspark.driver.bindAddress=172.17.0.2 > org.apache.spark.examples.SparkPi > Error: Could not find or load main class org.apache.spark.examples.SparkPi > > Found this stackoverflow question > https://stackoverflow.com/questions/49331570/spark-2-3-minikube-kubernetes-windows-demo-sparkpi-not-found > but there is no answer. > I also checked container file system, it contains > /opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar > > > 2018-04-11 1:17 GMT+08:00 Yinan Li <liyinan...@gmail.com>: >> >> The example jar path should be >> local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar. >> >> On Tue, Apr 10, 2018 at 1:34 AM, Dmitry <frostb...@gmail.com> wrote: >>> >>> Hello spent a lot of time to find what I did wrong , but not found. >>> I have a minikube WIndows based cluster ( Hyper V as hypervisor ) and try >>> to run examples against Spark 2.3. Tried several docker images builds: >>> * several builds that I build myself >>> * andrusha/spark-k8s:2.3.0-hadoop2.7 from docker hub >>> But when I try to submit job driver log returns class not found >>> exception >>> org.apache.spark.examples.SparkPi >>> >>> spark-submit --master k8s://https://ip:8443 --deploy-mode cluster >>> --name spark-pi --class org.apache.spark.examples.SparkPi --conf >>> spark.executor.instances=1 --executor-memory 1G --conf spark.kubernete >>> s.container.image=andrusha/spark-k8s:2.3.0-hadoop2.7 >>> local:///opt/spark/examples/spark-examples_2.11-2.3.0.jar >>> >>> I tried to use https://github.com/apache-spark-on-k8s/spark fork and it >>> is works without problems, more complex examples work also. >> >> > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
