UGI is the heart of darkness, but I've looked at at. Oracle have taken the scariest class in the codebase and are now adding jdk-version specfic reflection-based codepaths. Joy.
On Tue, 23 Jun 2026 at 04:06, Cheng Pan <[email protected]> wrote: > In addition, the Spark DT framework does not work correctly in Java 25 > (especially Spark on K8s mode) due JDK 22+ Subject propagation change. > HADOOP-19668[1] partially solves the issue but does not fully meet Spark > requirements, I proposed a simple approach HADOOP-19906[2][3] as an > alternative, hoping to get reviews by folks who know UGI stuff well. > > [1] https://issues.apache.org/jira/browse/HADOOP-19668 > [2] https://issues.apache.org/jira/browse/HADOOP-19906 > [3] https://github.com/apache/hadoop/pull/8522 > > Thanks, > Cheng Pan > > > > On Jun 23, 2026, at 10:48, Cheng Pan <[email protected]> wrote: > > I think this SPIP functionality has overlaps with [1] - both SPIPs solve > application-level credentials, and I’d like to know if you have plan to > extend this SPIP to support session-level credentials in the future. As a > reference, Kousuke Saruta and I had some high-level discussions on this in > [2]. > > [1] https://lists.apache.org/thread/cy2wqt9xtt5fh0qwmsfx9ht762cqpqs1 > [2] > https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?disco=AAAB88lRrIA > > Thanks, > Cheng Pan > > > > On Jun 23, 2026, at 02:52, Steve Loughran <[email protected]> wrote: > > yeah, hadoop dt interface doesn't say anything about kerberos being > required, it is just that spark doesn't ask for dt unless security is > enabled. s3a and abfs connectors will, if delegation tokens are enabled for > them, happily issue their tokens > > HadoopDelegationTokenProvider *does not require kerberos*. it is just that > spark doesn't ask filesystems for tokens without it. > > You can see this with the fetchdt command of cloudstore > https://github.com/steveloughran/cloudstore > > https://github.com/steveloughran/cloudstore/blob/main/src/site/markdown/fetchdt.md > > point it an FS and it'll ask for them; > > https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html > > azure can do the same with fs.azure.enable.delegation.token and a token > type, such as oauth. > > for example, s3a set to ask for session tokens in an extra restricted role > <property> > <name>fs.s3a.delegation.token.role.arn</name> > <value>${ARN-restricted}</value> > </property> > > <property> > <name>fs.s3a.delegation.token.binding</name> > > <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value> > </property> > > > then you can use cloudstore (which is soon to have its first asf release, > but currently needs to be downloaded from > https://github.com/steveloughran/cloudstore ) > > > bin/hadoop jar $CLOUDSTORE fetchdt out.tokens s3a://stevel-london/ > > Collecting tokens for 1 filesystem to to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > 2026-06-22 19:39:30,652 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:<init>(84)) - Starting: Fetching tokens for > s3a://stevel-london/ > 2026-06-22 19:39:32,204 [main] INFO delegation.S3ADelegationTokens > (DurationInfo.java:<init>(77)) - Starting: Creating New Delegation Token > 2026-06-22 19:39:32,282 [main] INFO auth.STSClientFactory > (STSClientFactory.java:lambda$requestSessionCredentials$0(227)) - > Requesting Amazon STS Session credentials > 2026-06-22 19:39:32,936 [main] INFO delegation.S3ADelegationTokens > (S3ADelegationTokens.java:noteTokenCreated(443)) - Created S3A Delegation > Token: Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, > Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; > uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; > encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on > VXM63P4JG2/192.168.50.99 at time 2026-06-22T18:39:32.212268Z.}; session > credentials, expiry 2026-06-23T06:39:32Z; (valid)) > 2026-06-22 19:39:32,938 [main] INFO delegation.S3ADelegationTokens > (DurationInfo.java:close(98)) - Creating New Delegation Token: duration > 0:00.732s > *Fetched token: Kind: S3ADelegationToken/Session, Service: > s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; > uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; > encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on > VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time > 2026-06-22T18:39:32.212268Z.}; session credentials, expiry > 2026-06-23T06:39:32Z; (valid))* > 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:close(190)) - Duration of Fetching tokens for > s3a://stevel-london/: 00:00:02.290 > 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:<init>(84)) - Starting: Saving 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > 2026-06-22 19:39:33,233 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:close(190)) - Duration of Saving 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens: 00:00:00.292 > Saved 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > > Token issued; no kerberos around and the file now has session credentials > valid for 12h and fs encryption settings included. > > Accordingly, I'm going to suggest a different design > > > 1. Don't bother with a new subclass > 2. Simply add a switch to enable token collection even if kerberos is > off > 3. For a test, add a subclass of file:// with a new url and see if you > can issue tokens off it. More rigorously, add a switch enabling it to blow > up during creation/unmarshalling to test spark reslience > > The hard part here is actually implementing any test DT implementation; if > you can avoid that your life is better. > > On Thu, 18 Jun 2026 at 11:55, Peter Toth <[email protected]> wrote: > >> Hi Parth, >> >> Thanks for the SPIP and for answering our questions. >> Does anyone have any other questions or points they'd like to discuss? >> >> Peter >> >> On Thu, Jun 4, 2026 at 10:02 PM Parth Chandra <[email protected]> wrote: >> >>> Hi all, >>> I've a proposal to enhance the current mechanism to distribute >>> delegation tokens and other secure tokens. >>> >>> The summary is that the current mechanism is gated behind Kerberos >>> even though the actual distribution does not require Kerberos except where >>> the tokens themselves are Kerberos tokens. Cloud environments may not have >>> a Kerberos setup and this creates an unnecessary setup step that users may >>> have to perform. The current implementation of KafkaDelegationTokenProvider >>> illustrates this. The implementation does not require Kerberos, yet it has >>> to pass the Kerberos gates. >>> >>> The proposal then is to allow a second path that does not require the >>> Kerberos gates unless the provider indicates that it be required. the >>> design has minimal change to the existing code and is fully backward >>> compatible. >>> >>> The proposal and corresponding JIRA are in [1], [2] >>> >>> I'd greatly appreciate it if committers can take some time to review >>> and provide feedback >>> >>> Thanks >>> >>> Parth >>> [1] >>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl >>> [2] https://issues.apache.org/jira/browse/SPARK-57252 >>> >> > >
