Hi Cheng Pan, There was an initial discussion on the overlap between the two and then we decided to keep them independent [1], so no there is no plan in this SPIP to get into session level support. The main purpose of this is to remove Kerberos gates which currently block the distribution of credentials that are not dependent on Kerberos while keeping the code change to a minimum.
[1] https://lists.apache.org/thread/yj31mhxohv36hxgvtvq6b9wy2yo51795 On Mon, Jun 22, 2026 at 7:50 PM Cheng Pan <[email protected]> wrote: > I think this SPIP functionality has overlaps with [1] - both SPIPs solve > application-level credentials, and I’d like to know if you have plan to > extend this SPIP to support session-level credentials in the future. As a > reference, Kousuke Saruta and I had some high-level discussions on this in > [2]. > > [1] https://lists.apache.org/thread/cy2wqt9xtt5fh0qwmsfx9ht762cqpqs1 > [2] > https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?disco=AAAB88lRrIA > > Thanks, > Cheng Pan > > > > On Jun 23, 2026, at 02:52, Steve Loughran <[email protected]> wrote: > > yeah, hadoop dt interface doesn't say anything about kerberos being > required, it is just that spark doesn't ask for dt unless security is > enabled. s3a and abfs connectors will, if delegation tokens are enabled for > them, happily issue their tokens > > HadoopDelegationTokenProvider *does not require kerberos*. it is just that > spark doesn't ask filesystems for tokens without it. > > You can see this with the fetchdt command of cloudstore > https://github.com/steveloughran/cloudstore > > https://github.com/steveloughran/cloudstore/blob/main/src/site/markdown/fetchdt.md > > point it an FS and it'll ask for them; > > https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html > > azure can do the same with fs.azure.enable.delegation.token and a token > type, such as oauth. > > for example, s3a set to ask for session tokens in an extra restricted role > <property> > <name>fs.s3a.delegation.token.role.arn</name> > <value>${ARN-restricted}</value> > </property> > > <property> > <name>fs.s3a.delegation.token.binding</name> > > <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value> > </property> > > > then you can use cloudstore (which is soon to have its first asf release, > but currently needs to be downloaded from > https://github.com/steveloughran/cloudstore ) > > > bin/hadoop jar $CLOUDSTORE fetchdt out.tokens s3a://stevel-london/ > > Collecting tokens for 1 filesystem to to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > 2026-06-22 19:39:30,652 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:<init>(84)) - Starting: Fetching tokens for > s3a://stevel-london/ > 2026-06-22 19:39:32,204 [main] INFO delegation.S3ADelegationTokens > (DurationInfo.java:<init>(77)) - Starting: Creating New Delegation Token > 2026-06-22 19:39:32,282 [main] INFO auth.STSClientFactory > (STSClientFactory.java:lambda$requestSessionCredentials$0(227)) - > Requesting Amazon STS Session credentials > 2026-06-22 19:39:32,936 [main] INFO delegation.S3ADelegationTokens > (S3ADelegationTokens.java:noteTokenCreated(443)) - Created S3A Delegation > Token: Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, > Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; > uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; > encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on > VXM63P4JG2/192.168.50.99 at time 2026-06-22T18:39:32.212268Z.}; session > credentials, expiry 2026-06-23T06:39:32Z; (valid)) > 2026-06-22 19:39:32,938 [main] INFO delegation.S3ADelegationTokens > (DurationInfo.java:close(98)) - Creating New Delegation Token: duration > 0:00.732s > *Fetched token: Kind: S3ADelegationToken/Session, Service: > s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; > uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; > encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on > VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time > 2026-06-22T18:39:32.212268Z.}; session credentials, expiry > 2026-06-23T06:39:32Z; (valid))* > 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:close(190)) - Duration of Fetching tokens for > s3a://stevel-london/: 00:00:02.290 > 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:<init>(84)) - Starting: Saving 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > 2026-06-22 19:39:33,233 [main] INFO commands.FetchTokens > (StoreDurationInfo.java:close(190)) - Duration of Saving 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens: 00:00:00.292 > Saved 1 token to > file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens > > Token issued; no kerberos around and the file now has session credentials > valid for 12h and fs encryption settings included. > > Accordingly, I'm going to suggest a different design > > > 1. Don't bother with a new subclass > 2. Simply add a switch to enable token collection even if kerberos is > off > 3. For a test, add a subclass of file:// with a new url and see if you > can issue tokens off it. More rigorously, add a switch enabling it to blow > up during creation/unmarshalling to test spark reslience > > The hard part here is actually implementing any test DT implementation; if > you can avoid that your life is better. > > On Thu, 18 Jun 2026 at 11:55, Peter Toth <[email protected]> wrote: > >> Hi Parth, >> >> Thanks for the SPIP and for answering our questions. >> Does anyone have any other questions or points they'd like to discuss? >> >> Peter >> >> On Thu, Jun 4, 2026 at 10:02 PM Parth Chandra <[email protected]> wrote: >> >>> Hi all, >>> I've a proposal to enhance the current mechanism to distribute >>> delegation tokens and other secure tokens. >>> >>> The summary is that the current mechanism is gated behind Kerberos >>> even though the actual distribution does not require Kerberos except where >>> the tokens themselves are Kerberos tokens. Cloud environments may not have >>> a Kerberos setup and this creates an unnecessary setup step that users may >>> have to perform. The current implementation of KafkaDelegationTokenProvider >>> illustrates this. The implementation does not require Kerberos, yet it has >>> to pass the Kerberos gates. >>> >>> The proposal then is to allow a second path that does not require the >>> Kerberos gates unless the provider indicates that it be required. the >>> design has minimal change to the existing code and is fully backward >>> compatible. >>> >>> The proposal and corresponding JIRA are in [1], [2] >>> >>> I'd greatly appreciate it if committers can take some time to review >>> and provide feedback >>> >>> Thanks >>> >>> Parth >>> [1] >>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl >>> [2] https://issues.apache.org/jira/browse/SPARK-57252 >>> >> >
