[
https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974621#comment-16974621
]
Jeremy edited comment on SPARK-29884 at 11/14/19 9:34 PM:
----------------------------------------------------------
After doing some debugging it seams like this might be in fabric k8s client. It
tries to use .kube/config even if it gets all the parameters is needs from
arguments.
was (Author: jeremyjjbrown):
After doing some debugging it seams like this might be in fabric k8s client. I
tries to use .kube/config even if it gets all the parameters is needs from
arguments.
> spark-submit to kuberentes can not parse valid ca certificate
> -------------------------------------------------------------
>
> Key: SPARK-29884
> URL: https://issues.apache.org/jira/browse/SPARK-29884
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.4
> Environment: A kuberentes cluster that has been in use for over 2
> years and handles large amounts of production payloads.
> Reporter: Jeremy
> Priority: Major
>
> spark submit can not be used to to schedule to kuberentes with oauth token
> and cacert
> {code:java}
> spark-submit \
> --deploy-mode cluster \
> --class org.apache.spark.examples.SparkPi \
> --master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
> --conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf
> spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
> \
> --conf spark.kubernetes.namespace=here-olp-3dds-sit \
> --conf spark.executor.instances=1 \
> --conf spark.app.name=spark-pi \
> --conf
> spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
> \
> --conf
> spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0
> \
> local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
> {code}
> returns
> {code:java}
> log4j:WARN No appenders could be found for logger
> (io.fabric8.kubernetes.client.Config).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main"
> io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
> ... 13 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 19 more
> {code}
> The cacert and token are both valid and work even with curl
> {code:java}
> curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H
> "Authorization: bearer $TOKEN" -v
> https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods
> -o out
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Trying 10.117.233.37:443...
> * TCP_NODELAY set
> * Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37)
> port 443 (#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
> CApath: none
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [58 bytes data]
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> { [1565 bytes data]
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> { [556 bytes data]
> * TLSv1.2 (IN), TLS handshake, Request CERT (13):
> { [588 bytes data]
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> { [4 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Certificate (11):
> } [7 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> } [37 bytes data]
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> } [16 bytes data]
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> { [16 bytes data]
> * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
> * ALPN, server accepted to use h2
> * Server certificate:
> * subject: CN=kubernetes-master
> * start date: Apr 2 13:30:06 2019 GMT
> * expire date: Apr 1 13:30:06 2020 GMT
> * subjectAltName: host "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com"
> matched cert's "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com"
> * issuer: C=DE; ST=Berlin; O=HERE Global BV; OU=OLP Engineering
> Infrastructure Development; CN=borg-dev-1-aws-eu-west-1;
> [email protected]
> * SSL certificate verify ok.
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade:
> len=0
> } [5 bytes data]
> * Using Stream ID: 1 (easy handle 0x558a2ede87b0)
> } [5 bytes data]
> > GET /api/v1/namespaces/here-olp-3dds-sit/pods HTTP/2
> > Host: api.borg-dev-1-aws-eu-west-1.k8s.in.here.com
> > User-Agent: curl/7.66.0
> > Accept: */*
> > Authorization: bearer *****************************************
> >
> { [5 bytes data]
> * Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
> } [5 bytes data]
> < HTTP/2 200
> < audit-id: a26869e9-7b42-4013-b1fa-839e963c6b78
> < content-type: application/json
> < date: Wed, 13 Nov 2019 20:20:18 GMT
> <
> { [5 bytes data]
> 100 56466 0 56466 0 0 64020 0 --:--:-- --:--:-- --:--:-- 63947
> * Connection #0 to host api.borg-dev-1-aws-eu-west-1.k8s.in.here.com left
> intact
> {code}
>
> {code:java}
> // code placeholder
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
