[jira] [Updated] (SPARK-29884) spark-submit to kuberentes can not parse valid ca certificate
[
https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy updated SPARK-29884:
---
Summary: spark-submit to kuberentes can not parse valid ca certificate
(was: spark-Submit to kuberentes can not parse valid ca certificate)
> spark-submit to kuberentes can not parse valid ca certificate
> -
>
> Key: SPARK-29884
> URL: https://issues.apache.org/jira/browse/SPARK-29884
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 2.4.4
> Environment: A kuberentes cluster that has been in use for over 2
> years and handles large amounts of production payloads.
>Reporter: Jeremy
>Priority: Major
>
> spark submit can not be used to to schedule to kuberentes with oauth token
> and cacert
> {code:java}
> spark-submit \
> --deploy-mode cluster \
> --class org.apache.spark.examples.SparkPi \
> --master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
> --conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf
> spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
> \
> --conf spark.kubernetes.namespace=here-olp-3dds-sit \
> --conf spark.executor.instances=1 \
> --conf spark.app.name=spark-pi \
> --conf
> spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
> \
> --conf
> spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0
> \
> local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
> {code}
> returns
> {code:java}
> log4j:WARN No appenders could be found for logger
> (io.fabric8.kubernetes.client.Config).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main"
> io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
> at
> org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
> at
> org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.security.cert.CertificateException: Could not parse
> certificate: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
> at
> java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
> ... 13 more
> Caused by: java.io.IOException: Empty input
> at
> sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
> ... 19 more
> {code}
> The cacert and token are both valid and work even with curl
> {code:java}
>
[jira] [Updated] (SPARK-29884) spark-Submit to kuberentes can not parse valid ca certificate
[
https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy updated SPARK-29884:
---
Description:
spark submit can not be used to to schedule to kuberentes with oauth token and
cacert
{code:java}
spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
--conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf
spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
\
--conf spark.kubernetes.namespace=here-olp-3dds-sit \
--conf spark.executor.instances=1 \
--conf spark.app.name=spark-pi \
--conf
spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
\
--conf
spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0
\
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
{code}
returns
{code:java}
log4j:WARN No appenders could be found for logger
(io.fabric8.kubernetes.client.Config).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at
io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
at
io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
at
io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
at
io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
... 13 more
Caused by: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 19 more
{code}
The cacert and token are both valid and work even with curl
{code:java}
curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H
"Authorization: bearer $TOKEN" -v
https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods
-o out
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying 10.117.233.37:443...
* TCP_NODELAY set
* Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37)
port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS
[jira] [Updated] (SPARK-29884) spark-Submit to kuberentes can not parse valid ca certificate
[
https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremy updated SPARK-29884:
---
Description:
spark submit can not be used to to schedule to kuberentes with oauth token and
cacert
{code:java}
spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \
--conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf
spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
\
--conf spark.kubernetes.namespace=here-olp-3dds-sit \
--conf spark.executor.instances=1 \
--conf spark.app.name=spark-pi \
--conf
spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0
\
--conf
spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0
\
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
{code}
returns
{code:java}
log4j:WARN No appenders could be found for logger
(io.fabric8.kubernetes.client.Config).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
Exception in thread "main"
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
at
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183)
at
org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
at
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.security.cert.CertificateException: Could not parse
certificate: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
at
java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at
io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104)
at
io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197)
at
io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
at
io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122)
at
io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78)
... 13 more
Caused by: java.io.IOException: Empty input
at
sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106)
... 19 more
{code}
The cacert and token are both valid and work even with curl
{code:java}
curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H
"Authorization: bearer $TOKEN" -v
https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods
-o out
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0*
Trying 10.117.233.37:443...
* TCP_NODELAY set
* Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37)
port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS
