To Parth > The DirectProviderPath proposed in this SPIP does not go through > Subject.doAs() or UserGroupInformation.doAs() and will be unaffected. The > existing Kerberos path will have to be updated.
Sorry, I overlook this reply. I think this is also affected. The JDK change breaks the Subject propagation between threads, that means you can not get the same Subject (UGI) instance from the task thread as the updateTokensTask thread, so you can not access any kind of the credential you offered from the task thread. Thanks, Cheng Pan > On Jun 25, 2026, at 11:40, Cheng Pan <[email protected]> wrote: > > To Parth > > > then we decided to keep them independent > > I see that, but if we decide to accept and implement both SPIPs, then we are > going to provide two approaches for users that enable cloud credentials > refresh, this is functionality redundant, and as you know, this part usually > involve private data and 3rd party services dependencies, when user report > issues, they are likely limited to share the related part of logs and > environment information to provide a minimal reproducible cases, this makes > diagnosis extremely difficult. Offering two distinct cloud credential refresh > mechanisms undoubtedly increases system complexity. > > I would lean towards to the OIDC Credential Propagation approach unless it > does not cover the functionality (user perspective) provided by this SPIP. > > > To Steve, > > Thanks for review the patch, I addressed your comments, and also would > encourage other reviewers/users to test it, especially for cases that run > Spark on K8s with JDK 25, and need to communicate to Kerberized services. > > Thanks, > Cheng Pan > > > >> On Jun 24, 2026, at 21:32, Steve Loughran <[email protected]> wrote: >> >> >> >> On Tue, 23 Jun 2026 at 23:36, Parth Chandra <[email protected] >> <mailto:[email protected]>> wrote: >>> Hi Cheng Pan, >>> >>> There was an initial discussion on the overlap between the two and then we >>> decided to keep them independent [1], so no there is no plan in this SPIP >>> to get into session level support. The main purpose of this is to remove >>> Kerberos gates which currently block the distribution of credentials that >>> are not dependent on Kerberos while keeping the code change to a minimum. >>> >>> >>> [1] https://lists.apache.org/thread/yj31mhxohv36hxgvtvq6b9wy2yo51795 >> >> you can just make that "require security" check optional. >> >> I've reviewed the Hadoop UGI PR. I can't believe how oracle have managed to >> make a class which was already scary even worse >> >>> >>> On Mon, Jun 22, 2026 at 7:50 PM Cheng Pan <[email protected] >>> <mailto:[email protected]>> wrote: >>>> I think this SPIP functionality has overlaps with [1] - both SPIPs solve >>>> application-level credentials, and I’d like to know if you have plan to >>>> extend this SPIP to support session-level credentials in the future. As a >>>> reference, Kousuke Saruta and I had some high-level discussions on this in >>>> [2]. >>>> >>>> [1] https://lists.apache.org/thread/cy2wqt9xtt5fh0qwmsfx9ht762cqpqs1 >>>> [2] >>>> https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?disco=AAAB88lRrIA >>>> >>>> Thanks, >>>> Cheng Pan >>>> >>>> >>>> >>>>> On Jun 23, 2026, at 02:52, Steve Loughran <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> yeah, hadoop dt interface doesn't say anything about kerberos being >>>>> required, it is just that spark doesn't ask for dt unless security is >>>>> enabled. s3a and abfs connectors will, if delegation tokens are enabled >>>>> for them, happily issue their tokens >>>>> >>>>> HadoopDelegationTokenProvider *does not require kerberos*. it is just >>>>> that spark doesn't ask filesystems for tokens without it. >>>>> >>>>> You can see this with the fetchdt command of cloudstore >>>>> https://github.com/steveloughran/cloudstore >>>>> https://github.com/steveloughran/cloudstore/blob/main/src/site/markdown/fetchdt.md >>>>> >>>>> point it an FS and it'll ask for them; >>>>> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html >>>>> >>>>> azure can do the same with fs.azure.enable.delegation.token and a token >>>>> type, such as oauth. >>>>> >>>>> for example, s3a set to ask for session tokens in an extra restricted role >>>>> <property> >>>>> <name>fs.s3a.delegation.token.role.arn</name> >>>>> <value>${ARN-restricted}</value> >>>>> </property> >>>>> >>>>> <property> >>>>> <name>fs.s3a.delegation.token.binding</name> >>>>> >>>>> <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value> >>>>> </property> >>>>> >>>>> >>>>> then you can use cloudstore (which is soon to have its first asf release, >>>>> but currently needs to be downloaded from >>>>> https://github.com/steveloughran/cloudstore ) >>>>> >>>>> > bin/hadoop jar $CLOUDSTORE fetchdt out.tokens s3a://stevel-london/ >>>>> >>>>> Collecting tokens for 1 filesystem to to >>>>> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >>>>> 2026-06-22 19:39:30,652 [main] INFO commands.FetchTokens >>>>> (StoreDurationInfo.java:<init>(84)) - Starting: Fetching tokens for >>>>> s3a://stevel-london/ >>>>> 2026-06-22 19:39:32,204 [main] INFO delegation.S3ADelegationTokens >>>>> (DurationInfo.java:<init>(77)) - Starting: Creating New Delegation Token >>>>> 2026-06-22 19:39:32,282 [main] INFO auth.STSClientFactory >>>>> (STSClientFactory.java:lambda$requestSessionCredentials$0(227)) - >>>>> Requesting Amazon STS Session credentials >>>>> 2026-06-22 19:39:32,936 [main] INFO delegation.S3ADelegationTokens >>>>> (S3ADelegationTokens.java:noteTokenCreated(443)) - Created S3A Delegation >>>>> Token: Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, >>>>> Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; >>>>> uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; >>>>> encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on >>>>> VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time >>>>> 2026-06-22T18:39:32.212268Z.}; session credentials, expiry >>>>> 2026-06-23T06:39:32Z; (valid)) >>>>> 2026-06-22 19:39:32,938 [main] INFO delegation.S3ADelegationTokens >>>>> (DurationInfo.java:close(98)) - Creating New Delegation Token: duration >>>>> 0:00.732s >>>>> Fetched token: Kind: S3ADelegationToken/Session, Service: >>>>> s3a://stevel-london, Ident: >>>>> (S3ATokenIdentifier{S3ADelegationToken/Session; uri=s3a://stevel-london; >>>>> timestamp=1782153572890; renewer=stevel; encryption=SSE-KMS; >>>>> 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on VXM63P4JG2/192.168.50.99 >>>>> <http://192.168.50.99/> at time 2026-06-22T18:39:32.212268Z.}; session >>>>> credentials, expiry 2026-06-23T06:39:32Z; (valid)) >>>>> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >>>>> (StoreDurationInfo.java:close(190)) - Duration of Fetching tokens for >>>>> s3a://stevel-london/: 00:00:02.290 >>>>> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >>>>> (StoreDurationInfo.java:<init>(84)) - Starting: Saving 1 token to >>>>> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >>>>> 2026-06-22 19:39:33,233 [main] INFO commands.FetchTokens >>>>> (StoreDurationInfo.java:close(190)) - Duration of Saving 1 token to >>>>> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens: 00:00:00.292 >>>>> Saved 1 token to >>>>> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >>>>> >>>>> Token issued; no kerberos around and the file now has session credentials >>>>> valid for 12h and fs encryption settings included. >>>>> >>>>> Accordingly, I'm going to suggest a different design >>>>> >>>>> Don't bother with a new subclass >>>>> Simply add a switch to enable token collection even if kerberos is off >>>>> For a test, add a subclass of file:// with a new url and see if you can >>>>> issue tokens off it. More rigorously, add a switch enabling it to blow up >>>>> during creation/unmarshalling to test spark reslience >>>>> The hard part here is actually implementing any test DT implementation; >>>>> if you can avoid that your life is better. >>>>> >>>>> On Thu, 18 Jun 2026 at 11:55, Peter Toth <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>>> Hi Parth, >>>>>> >>>>>> Thanks for the SPIP and for answering our questions. >>>>>> Does anyone have any other questions or points they'd like to discuss? >>>>>> >>>>>> Peter >>>>>> >>>>>> On Thu, Jun 4, 2026 at 10:02 PM Parth Chandra <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>>> Hi all, >>>>>>> I've a proposal to enhance the current mechanism to distribute >>>>>>> delegation tokens and other secure tokens. >>>>>>> >>>>>>> The summary is that the current mechanism is gated behind Kerberos >>>>>>> even though the actual distribution does not require Kerberos except >>>>>>> where the tokens themselves are Kerberos tokens. Cloud environments may >>>>>>> not have a Kerberos setup and this creates an unnecessary setup step >>>>>>> that users may have to perform. The current implementation of >>>>>>> KafkaDelegationTokenProvider illustrates this. The implementation does >>>>>>> not require Kerberos, yet it has to pass the Kerberos gates. >>>>>>> >>>>>>> The proposal then is to allow a second path that does not require the >>>>>>> Kerberos gates unless the provider indicates that it be required. the >>>>>>> design has minimal change to the existing code and is fully backward >>>>>>> compatible. >>>>>>> >>>>>>> The proposal and corresponding JIRA are in [1], [2] >>>>>>> >>>>>>> I'd greatly appreciate it if committers can take some time to review >>>>>>> and provide feedback >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Parth >>>>>>> [1] >>>>>>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl >>>>>>> [2] https://issues.apache.org/jira/browse/SPARK-57252 >>>> >
