[
https://issues.apache.org/jira/browse/SPARK-50555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andre Piwoni updated SPARK-50555:
---------------------------------
Description:
SPARK-33720 via [PR #30684|https://github.com/apache/spark/pull/30684] broke
usage of {{insecure-skip-tls-verify}} via KUBECONFIG file when using cluster
context with auto configure.
PR for this issue overwrites {{insecure-skip-tls-verify}} from KUBECONFIG set
by autoconfigure when new KUBERNETES_TRUST_CERTIFICATES property is missing
because it defaults to false. With this bug it is impossible to drive hostname
verification via KUBECONFIG when multiple clusters are present with different
requirements.
Correct implementation should do the following:
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=true then no hostname verification
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=false then do hostname verfication
# If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
I have verified autoconfigure with insecure-skip-tls-verify set to true works
in 3.1.3 before this change.
Relevant code from Spark and kuubernetes-client:
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
ConfigBuilder("spark.kubernetes.trust.certificates")
.createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
.withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
!= null && currentCluster.getInsecureSkipTlsVerify())
{code}
was:
SPARK-33720 via [PR #30684|https://github.com/apache/spark/pull/30684] broke
usage of `insecure-skip-tls-verify` via KUBECONFIG file when using cluster
context with auto configure.
PR for this issue overwrites {{insecure-skip-tls-verify}} from KUBECONFIG set
by autoconfigure when new KUBERNETES_TRUST_CERTIFICATES property is missing
because it defaults to false. With this bug it is impossible to drive hostname
verification via KUBECONFIG when multiple clusters are present with different
requirements.
Correct implementation should do the following:
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=true then no hostname verification
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=false then do hostname verfication
# If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
I have verified autoconfigure with insecure-skip-tls-verify set to true works
in 3.1.3 before this change.
Relevant code from Spark and kuubernetes-client:
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
ConfigBuilder("spark.kubernetes.trust.certificates")
.createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
.withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
!= null && currentCluster.getInsecureSkipTlsVerify())
{code}
> Broken support for disabling hostname verification based on
> insecure-skip-tls-verify in KUBECONFIG
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-50555
> URL: https://issues.apache.org/jira/browse/SPARK-50555
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.2.0
> Reporter: Andre Piwoni
> Priority: Major
>
> SPARK-33720 via [PR #30684|https://github.com/apache/spark/pull/30684] broke
> usage of {{insecure-skip-tls-verify}} via KUBECONFIG file when using cluster
> context with auto configure.
> PR for this issue overwrites {{insecure-skip-tls-verify}} from KUBECONFIG set
> by autoconfigure when new KUBERNETES_TRUST_CERTIFICATES property is missing
> because it defaults to false. With this bug it is impossible to drive
> hostname verification via KUBECONFIG when multiple clusters are present with
> different requirements.
> Correct implementation should do the following:
> # If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
> isTrustCerts=true then no hostname verification
> # If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
> isTrustCerts=false then do hostname verfication
> # If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
> I have verified autoconfigure with insecure-skip-tls-verify set to true works
> in 3.1.3 before this change.
> Relevant code from Spark and kuubernetes-client:
>
> {code:java}
> val KUBERNETES_TRUST_CERTIFICATES =
> ConfigBuilder("spark.kubernetes.trust.certificates")
> .createWithDefault(false)
> val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
> .withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
> // kubernetes-client Config.autoConfigure
> config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
> currentCluster.getInsecureSkipTlsVerify());
> config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
> != null && currentCluster.getInsecureSkipTlsVerify())
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
