On Tue, 23 Jun 2026 at 23:36, Parth Chandra <[email protected]> wrote:
> Hi Cheng Pan, > > There was an initial discussion on the overlap between the two and then we > decided to keep them independent [1], so no there is no plan in this SPIP > to get into session level support. The main purpose of this is to remove > Kerberos gates which currently block the distribution of credentials that > are not dependent on Kerberos while keeping the code change to a minimum. > > > [1] https://lists.apache.org/thread/yj31mhxohv36hxgvtvq6b9wy2yo51795 > you can just make that "require security" check optional. I've reviewed the Hadoop UGI PR. I can't believe how oracle have managed to make a class which was already scary even worse > > On Mon, Jun 22, 2026 at 7:50 PM Cheng Pan <[email protected]> wrote: > >> I think this SPIP functionality has overlaps with [1] - both SPIPs solve >> application-level credentials, and I’d like to know if you have plan to >> extend this SPIP to support session-level credentials in the future. As a >> reference, Kousuke Saruta and I had some high-level discussions on this in >> [2]. >> >> [1] https://lists.apache.org/thread/cy2wqt9xtt5fh0qwmsfx9ht762cqpqs1 >> [2] >> https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?disco=AAAB88lRrIA >> >> Thanks, >> Cheng Pan >> >> >> >> On Jun 23, 2026, at 02:52, Steve Loughran <[email protected]> wrote: >> >> yeah, hadoop dt interface doesn't say anything about kerberos being >> required, it is just that spark doesn't ask for dt unless security is >> enabled. s3a and abfs connectors will, if delegation tokens are enabled for >> them, happily issue their tokens >> >> HadoopDelegationTokenProvider *does not require kerberos*. it is just >> that spark doesn't ask filesystems for tokens without it. >> >> You can see this with the fetchdt command of cloudstore >> https://github.com/steveloughran/cloudstore >> >> https://github.com/steveloughran/cloudstore/blob/main/src/site/markdown/fetchdt.md >> >> point it an FS and it'll ask for them; >> >> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html >> >> azure can do the same with fs.azure.enable.delegation.token and a token >> type, such as oauth. >> >> for example, s3a set to ask for session tokens in an extra restricted role >> <property> >> <name>fs.s3a.delegation.token.role.arn</name> >> <value>${ARN-restricted}</value> >> </property> >> >> <property> >> <name>fs.s3a.delegation.token.binding</name> >> >> <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value> >> </property> >> >> >> then you can use cloudstore (which is soon to have its first asf release, >> but currently needs to be downloaded from >> https://github.com/steveloughran/cloudstore ) >> >> > bin/hadoop jar $CLOUDSTORE fetchdt out.tokens s3a://stevel-london/ >> >> Collecting tokens for 1 filesystem to to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> 2026-06-22 19:39:30,652 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:<init>(84)) - Starting: Fetching tokens for >> s3a://stevel-london/ >> 2026-06-22 19:39:32,204 [main] INFO delegation.S3ADelegationTokens >> (DurationInfo.java:<init>(77)) - Starting: Creating New Delegation Token >> 2026-06-22 19:39:32,282 [main] INFO auth.STSClientFactory >> (STSClientFactory.java:lambda$requestSessionCredentials$0(227)) - >> Requesting Amazon STS Session credentials >> 2026-06-22 19:39:32,936 [main] INFO delegation.S3ADelegationTokens >> (S3ADelegationTokens.java:noteTokenCreated(443)) - Created S3A Delegation >> Token: Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, >> Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; >> uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; >> encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on >> VXM63P4JG2/192.168.50.99 at time 2026-06-22T18:39:32.212268Z.}; session >> credentials, expiry 2026-06-23T06:39:32Z; (valid)) >> 2026-06-22 19:39:32,938 [main] INFO delegation.S3ADelegationTokens >> (DurationInfo.java:close(98)) - Creating New Delegation Token: duration >> 0:00.732s >> *Fetched token: Kind: S3ADelegationToken/Session, Service: >> s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; >> uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; >> encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on >> VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time >> 2026-06-22T18:39:32.212268Z.}; session credentials, expiry >> 2026-06-23T06:39:32Z; (valid))* >> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:close(190)) - Duration of Fetching tokens for >> s3a://stevel-london/: 00:00:02.290 >> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:<init>(84)) - Starting: Saving 1 token to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> 2026-06-22 19:39:33,233 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:close(190)) - Duration of Saving 1 token to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens: 00:00:00.292 >> Saved 1 token to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> >> Token issued; no kerberos around and the file now has session credentials >> valid for 12h and fs encryption settings included. >> >> Accordingly, I'm going to suggest a different design >> >> >> 1. Don't bother with a new subclass >> 2. Simply add a switch to enable token collection even if kerberos is >> off >> 3. For a test, add a subclass of file:// with a new url and see if >> you can issue tokens off it. More rigorously, add a switch enabling it to >> blow up during creation/unmarshalling to test spark reslience >> >> The hard part here is actually implementing any test DT implementation; >> if you can avoid that your life is better. >> >> On Thu, 18 Jun 2026 at 11:55, Peter Toth <[email protected]> wrote: >> >>> Hi Parth, >>> >>> Thanks for the SPIP and for answering our questions. >>> Does anyone have any other questions or points they'd like to discuss? >>> >>> Peter >>> >>> On Thu, Jun 4, 2026 at 10:02 PM Parth Chandra <[email protected]> wrote: >>> >>>> Hi all, >>>> I've a proposal to enhance the current mechanism to distribute >>>> delegation tokens and other secure tokens. >>>> >>>> The summary is that the current mechanism is gated behind Kerberos >>>> even though the actual distribution does not require Kerberos except where >>>> the tokens themselves are Kerberos tokens. Cloud environments may not have >>>> a Kerberos setup and this creates an unnecessary setup step that users may >>>> have to perform. The current implementation of KafkaDelegationTokenProvider >>>> illustrates this. The implementation does not require Kerberos, yet it has >>>> to pass the Kerberos gates. >>>> >>>> The proposal then is to allow a second path that does not require the >>>> Kerberos gates unless the provider indicates that it be required. the >>>> design has minimal change to the existing code and is fully backward >>>> compatible. >>>> >>>> The proposal and corresponding JIRA are in [1], [2] >>>> >>>> I'd greatly appreciate it if committers can take some time to review >>>> and provide feedback >>>> >>>> Thanks >>>> >>>> Parth >>>> [1] >>>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl >>>> [2] https://issues.apache.org/jira/browse/SPARK-57252 >>>> >>> >>
