Zoltan Chovan has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24654
Change subject: [spark] Add delegation-token support for the Spark connector ...................................................................... [spark] Add delegation-token support for the Spark connector Enable the Kudu Spark connector to work in secure clusters where the Spark driver has no Kerberos ticket -- notably YARN cluster mode with --proxy-user and no keytab, where the remote driver starts on a NodeManager with no TGT and therefore cannot export Kudu credentials itself. Spark 3.5 exposes org.apache.spark.security.HadoopDelegationTokenProvider, discovered via the Java ServiceLoader and invoked at submit time while the submitter still holds Kerberos credentials. This change implements that extension point for Kudu and teaches KuduContext to consume the resulting credentials at runtime: - KuduSparkSecurity: shared constants (spark.kudu.master, an optional spark.kudu.saslProtocolName, the kudu.authn.credentials alias) and helpers to store/read the Kudu credential bytes in Hadoop Credentials / UserGroupInformation. - KuduDelegationTokenProvider (serviceName "kudu"): when Hadoop security is on and spark.kudu.master is set, builds a short-lived KuduClient with the submitter's credentials, exports the Kudu authentication credentials, and stores them in the Hadoop Credentials bundle that Spark/YARN ships to the driver and executors. It honors spark.kudu.saslProtocolName for clusters whose Kudu service principal is not the default "kudu". Registered via META-INF/services so Spark discovers it without a Spark patch, and disabled by the standard spark.security.credentials.kudu.enabled=false switch. It performs no token renewal (Kudu authn credentials are not modeled as renewable Hadoop delegation tokens), so long-running applications remain bounded by the Kudu authn-token lifetime. - KuduContext: initialize authnCredentials from KuduSparkSecurity.getCredentialsFromUGI() when present, falling back to the existing driver-side export otherwise. In cluster mode with --proxy-user the remote driver has no TGT, so the shipped credentials are the only way to authenticate; client/keytab deployments keep working via the fallback. The existing executor-side importAuthenticationCredentials propagation is unchanged. Tests: - KuduDelegationTokenProviderTest (no cluster): ServiceLoader discovery, delegationTokensRequired when Hadoop security is off, and a KuduSparkSecurity round-trip. - KuduDelegationTokenProviderSecurityTest (secure mini cluster): the delegationTokensRequired truth table, a real obtain-and-store, and an obtain against a cluster using a non-default SASL protocol name. - KuduContextUgiCredentialsTest (no cluster): KuduContext prefers UGI credentials without attempting a driver-side export. - KuduContextTest.testFallsBackToExportWhenUgiEmpty: the fallback path. - org.apache.spark.security.KuduSparkSubmitTokenTest (no cluster) and KuduSparkSubmitTokenSecurityTest (secure mini cluster): drive Spark's real HadoopDelegationTokenManager to confirm the provider is discovered/invoked (and the disable switch honored), then exercise the full submit -> Credentials -> UGI -> KuduContext read/write path. Also add a verify_jars.pl allowlist entry for the new META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider file. Change-Id: Iacda7b30fd1de3b91301774147bfa8706cefc17f --- M build-support/verify_jars.pl A java/kudu-spark/src/main/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala A java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProvider.scala A java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduSparkSecurity.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala A java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextUgiCredentialsTest.scala A java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProviderSecurityTest.scala A java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProviderTest.scala A java/kudu-spark/src/test/scala/org/apache/spark/security/KuduSparkSubmitTokenSecurityTest.scala A java/kudu-spark/src/test/scala/org/apache/spark/security/KuduSparkSubmitTokenTest.scala 11 files changed, 745 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/24654/1 -- To view, visit http://gerrit.cloudera.org:8080/24654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iacda7b30fd1de3b91301774147bfa8706cefc17f Gerrit-Change-Number: 24654 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Chovan <[email protected]>
