Hello Marton Greber, Alexey Serbin, Zoltan Martonka, Ashwani Raina, Attila
Bukor, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24654
to look at the new patch set (#3).
Change subject: [spark] Add delegation-token support for the Spark connector
......................................................................
[spark] Add delegation-token support for the Spark connector
Enable the Kudu Spark connector to work in secure clusters where the
Spark driver has no Kerberos ticket -- notably YARN cluster mode with
--proxy-user and no keytab, where the remote driver starts on a
NodeManager with no TGT and therefore cannot export Kudu credentials
itself.
Spark 3.5 exposes org.apache.spark.security.HadoopDelegationTokenProvider,
discovered via the Java ServiceLoader and invoked at submit time while
the submitter still holds Kerberos credentials. This change implements
that extension point for Kudu and teaches KuduContext to consume the
resulting credentials at runtime:
- KuduSparkSecurity: shared constants (spark.kudu.master, an optional
spark.kudu.saslProtocolName, the kudu.authn.credentials alias) and
helpers to store/read the Kudu credential bytes in Hadoop Credentials /
UserGroupInformation.
- KuduDelegationTokenProvider (serviceName "kudu"): when Hadoop security
is on and spark.kudu.master is set, builds a short-lived KuduClient
with the submitter's credentials, exports the Kudu authentication
credentials, and stores them in the Hadoop Credentials bundle that
Spark/YARN ships to the driver and executors. It honors
spark.kudu.saslProtocolName for clusters whose Kudu service principal is
not the default "kudu". Registered via META-INF/services so Spark
discovers it without a Spark patch, and disabled by the standard
spark.security.credentials.kudu.enabled=false switch. It performs no
token renewal (Kudu authn credentials are not modeled as renewable
Hadoop delegation tokens), so long-running applications remain bounded
by the Kudu authn-token lifetime.
- KuduContext: initialize authnCredentials from
KuduSparkSecurity.getCredentialsFromUGI() when present, falling back
to the existing driver-side export otherwise. In cluster mode with
--proxy-user the remote driver has no TGT, so the shipped credentials
are the only way to authenticate; client/keytab deployments keep
working via the fallback. The existing executor-side
importAuthenticationCredentials propagation is unchanged.
Tests:
- KuduDelegationTokenProviderTest (no cluster): ServiceLoader discovery,
delegationTokensRequired when Hadoop security is off, and a
KuduSparkSecurity round-trip.
- KuduDelegationTokenProviderSecurityTest (secure mini cluster): the
delegationTokensRequired truth table, a real obtain-and-store, and an
obtain against a cluster using a non-default SASL protocol name.
- KuduContextUgiCredentialsTest (no cluster): KuduContext prefers UGI
credentials without attempting a driver-side export.
- KuduContextTest.testFallsBackToExportWhenUgiEmpty: the fallback path.
- org.apache.spark.security.KuduSparkSubmitTokenTest (no cluster) and
KuduSparkSubmitTokenSecurityTest (secure mini cluster): drive Spark's
real HadoopDelegationTokenManager to confirm the provider is
discovered/invoked (and the disable switch honored), then exercise the
full submit -> Credentials -> UGI -> KuduContext read/write path.
Also add a verify_jars.pl allowlist entry for the new
META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
file.
Change-Id: Iacda7b30fd1de3b91301774147bfa8706cefc17f
---
M build-support/verify_jars.pl
M java/config/spotbugs/excludeFilter.xml
A
java/kudu-spark/src/main/resources/META-INF/services/org.apache.spark.security.HadoopDelegationTokenProvider
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
A
java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProvider.scala
A
java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduSparkSecurity.scala
M
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
A
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextUgiCredentialsTest.scala
A
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProviderSecurityTest.scala
A
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduDelegationTokenProviderTest.scala
A
java/kudu-spark/src/test/scala/org/apache/spark/security/KuduSparkSubmitTokenSecurityTest.scala
A
java/kudu-spark/src/test/scala/org/apache/spark/security/KuduSparkSubmitTokenTest.scala
12 files changed, 754 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/24654/3
--
To view, visit http://gerrit.cloudera.org:8080/24654
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iacda7b30fd1de3b91301774147bfa8706cefc17f
Gerrit-Change-Number: 24654
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Chovan <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Attila Bukor <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Zoltan Chovan <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
