Andre Piwoni created SPARK-50555:
------------------------------------
Summary: Broken support for disabling hostname verification based
on insecure-skip-tls-verify in KUBECONFIG
Key: SPARK-50555
URL: https://issues.apache.org/jira/browse/SPARK-50555
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.2.0
Reporter: Andre Piwoni
SPARK-33720 via [PR #30684|https://github.com/apache/spark/pull/30684] broke
usage of `insecure-skip-tls-verify` via KUBECONFIG file when using cluster
context with auto configure.
PR for this issue overwrites {{insecure-skip-tls-verify}} from KUBECONFIG set
by autoconfigure when new KUBERNETES_TRUST_CERTIFICATES property is missing
because it defaults to false. With this bug it is impossible to drive hostname
verification via KUBECONFIG when multiple clusters are present with different
requirements.
Correct implementation should do the following:
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=true then no hostname verification
# If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure
isTrustCerts=false then do hostname verfication
# If KUBERNETES_TRUST_CERTIFICATES is set use this as an override
I have verified autoconfigure with insecure-skip-tls-verify set to true works
in 3.1.3 before this change.
Relevant code from Spark and kuubernetes-client:
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
ConfigBuilder("spark.kubernetes.trust.certificates")
.createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
.withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null &&
currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify()
!= null && currentCluster.getInsecureSkipTlsVerify())
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
