Can you please paste the full exception trace, and mention spark and k8s version?
On Mon, Jan 4, 2021 at 6:19 PM Sachit Murarka <[email protected]> wrote: > Hi Prashant > > Thanks for the response! > > I created the service account with the permissions and following is the > command: > > spark-submit --deploy-mode cluster --master k8s://http://ip:port --name > "sachit" --conf spark.kubernetes.pyspark.pythonVersion=3 > --conf spark.kubernetes.namespace=spark-test --conf > spark.executor.instances=5 --conf > spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf > spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py > > Kind Regards, > Sachit Murarka > > > On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma <[email protected]> > wrote: > >> Hi Sachit, >> >> Can you give more details on how did you run? i.e. spark submit command. >> My guess is, a service account with sufficient privilege is not provided. >> Please see: >> http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac >> >> Thanks, >> >> On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <[email protected]> >> wrote: >> >>> Hi All, >>> I am getting the below error when I am trying to run the spark job on >>> Kubernetes, I am running it in cluster mode. >>> >>> Exception in thread "main" >>> io.fabric8.kubernetes.client.KubernetesClientException: Operation: >>> [create] for kind: [Pod] with name: [null] in namespace: >>> [spark-test] failed. >>> at >>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) >>> at >>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) >>> >>> >>> I saw there was an JIRA opened already. SPARK-31786 >>> I tried the parameter mentioned in JIRA >>> too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work. >>> Can anyone suggest what can be done? >>> >>> Kind Regards, >>> Sachit Murarka >>> >>
