Siddaraju G C created SPARK-44573:
-------------------------------------
Summary: Couldn't submit Spark application to Kubenetes in
versions v1.27.3
Key: SPARK-44573
URL: https://issues.apache.org/jira/browse/SPARK-44573
Project: Spark
Issue Type: Bug
Components: Kubernetes, Spark Submit
Affects Versions: 3.4.1
Reporter: Siddaraju G C
Spark-submit ( cluster mode on Kubernetes ) results error
*io.fabric8.kubernetes.client.KubernetesClientException* on my 3 nodes k8s
cluster.
Steps followed:
* using IBM cloud, created 3 Instances
* 1st Instance act as master node and another two acts as worker nodes
{noformat}
root@vsi-spark-master:/opt# kubectl get nodes
NAME STATUS ROLES AGE VERSION
vsi-spark-master Ready control-plane,master 2d v1.27.3+k3s1
vsi-spark-worker-1 Ready <none> 47h v1.27.3+k3s1
vsi-spark-worker-2 Ready <none> 47h
v1.27.3+k3s1{noformat}
* Copy spark-3.4.1-bin-hadoop3.tgz in to /opt/spark folder
* Ran spark by using below command
{noformat}
root@vsi-spark-master:/opt# /opt/spark/bin/spark-submit --master
k8s://http://<master_node_IP>:6443 --conf
spark.kubernetes.authenticate.submission.oauthToken=$TOKEN --deploy-mode
cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf
spark.executor.instances=5 --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf
spark.kubernetes.container.image=sushmakorati/testrepo:pyrandomGB
local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar{noformat}
* And getting below error message.
{noformat}
3/07/27 12:56:26 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS.
23/07/27 12:56:26 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
23/07/27 12:56:26 INFO SparkKubernetesClientFactory: Auto-configuring K8S
client using current context from users K8S config file
23/07/27 12:56:26 INFO KerberosConfDriverFeatureStep: You have not specified a
krb5.conf file locally or via a ConfigMap. Make sure that you have the
krb5.conf locally on the driver image.
23/07/27 12:56:27 ERROR Client: Please check "kubectl auth can-i create pod"
first. It should be yes.
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
at
io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1113)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:93)
at
org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:153)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$5(KubernetesClientApplication.scala:250)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$5$adapted(KubernetesClientApplication.scala:244)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2786)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:244)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:216)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Connection reset
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:535)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:558)
at
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:349)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:711)
at
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93)
at
io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
... 15 more
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at okio.Okio$2.read(Okio.java:140)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.read(RealBufferedSource.java:47)
at
okhttp3.internal.http1.Http1Codec$AbstractSource.read(Http1Codec.java:363)
at
okhttp3.internal.http1.Http1Codec$UnknownLengthSource.read(Http1Codec.java:507)
at okio.RealBufferedSource.exhausted(RealBufferedSource.java:57)
at
io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:127)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
23/07/27 12:56:27 INFO ShutdownHookManager: Shutdown hook called
23/07/27 12:56:27 INFO ShutdownHookManager: Deleting directory
/tmp/spark-70ee50ef-d9e9-4220-91f4-15a282031095{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
