Thanks so much! I'll take a look at the guide right now. The versions should all be 2.2 of spark. In my configuration, I'm using
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \ --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \ and I built the version of spark located here: https://github.com/apache-spark-on-k8s/spark which also seems to be 2.2 On Mon, Apr 30, 2018 at 12:09 PM, Yinan Li <liyinan...@gmail.com> wrote: > Which version of Spark are you using to run spark-submit, and which > version of Spark your container image is based off? This looks to be caused > my mismatched versions of Spark used for spark-submit and for the > driver/executor at runtime. > > On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <hol...@pigscanfly.ca> > wrote: > >> So, while its not perfect, I have a guide focused on running custom Spark >> on GKE https://cloud.google.com/blog/big-data/2018/03/testing-f >> uture-apache-spark-releases-and-changes-on-google-kubernetes >> -engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE >> there is a solutions article - https://cloud.google.com/sol >> utions/spark-on-kubernetes-engine which could be relevant. >> >> On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <ericswang1...@gmail.com> >> wrote: >> >>> Hello all, >>> >>> I've been trying to spark-submit a job to the Google Kubernetes Engine >>> but I keep encountering a "Exception in thread "main" >>> java.lang.IllegalArgumentException: Server properties file given at >>> /opt/spark/work-dir/driver does not exist or is not a file." >>> error. I'm unsure of how to even begin debugging this so any help would >>> be greatly appreciated. I've attached the logs and the full spark-submit >>> command I'm running here: https://gist.github.com/ >>> erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed >>> >>> For reference, I've been following this guide: https://apache-spark-on >>> -k8s.github.io/userdocs/running-on-kubernetes.html >>> >>> Thanks, >>> Eric >>> >>> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> > >
