[
https://issues.apache.org/jira/browse/SPARK-33720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17904882#comment-17904882
]
Andre Piwoni edited comment on SPARK-33720 at 12/11/24 5:58 PM:
----------------------------------------------------------------
This resolved one issue and created another one because it breaks usage of
`insecure-skip-tls-verify` via KUBECONFIG file when using cluster context with
auto configure.
Code below basically overwrites `insecure-skip-tls-verify` from KUBECONFIG set
by autoconfigure when KUBERNETES_TRUST_CERTIFICATES is missing because it
defaults to false. With this bug it is impossible to drive hostname
verification via KUBECONFIG when multiple clusters are present with different
requirements.
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
ConfigBuilder("spark.kubernetes.trust.certificates")
.createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
.withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
!= null && currentCluster.getInsecureSkipTlsVerify())
{code}
Correct implementation should do the following:
1. If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=true then no hostname verification
2. If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=false then do hostname verfication
3. If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
This is broken starting with version 3.2. I have verified autoconfigure with
insecure-skip-tls-verify set to true works in 3.1.3 before this change.
was (Author: apiwoni):
This resolved one issue and created another one because it breaks usage of
`insecure-skip-tls-verify` via KUBECONFIG file when using cluster context with
auto configure.
Code below basically overwrites `insecure-skip-tls-verify` from KUBECONFIG set
by autoconfigure when KUBERNETES_TRUST_CERTIFICATES is missing because it
defaults to false. With this bug it is impossible to drive hostname
verification via KUBECONFIG when multiple clusters are present with different
requirements.
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
ConfigBuilder("spark.kubernetes.trust.certificates")
.createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
.withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
!= null && currentCluster.getInsecureSkipTlsVerify())
{code}
Correct implementation should do the following:
1. If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=true then no hostname verification
2. If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=false then do hostname verfication
3. If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
> Support submit to k8s only with token
> -------------------------------------
>
> Key: SPARK-33720
> URL: https://issues.apache.org/jira/browse/SPARK-33720
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.2.0
> Reporter: hong dongdong
> Assignee: hong dongdong
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
