Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzip wrote: > So, this is purely about handling Hadoop authentication for Hadoop services. This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's : unless a) we have explicitly based our support on it, or b) generalized it sufficiently that we can support others, or c) keep it an impl detail in core (but exposed in yarn for backward compatibility). IMO (a) or (b) require a dev@ discussion. Until now, this (hadoop security) support was restricted to yarn in spark (with a couple of minor other uses iirc). @mgummelt: > The only thing the new ServiceCredentialProvider interface enforces is that the credentials must be added to a Credentials object, which is a hadoop class. <snip> The spi makes assumptions about the environment within which the credential provider is invoked, how the tokens are updated at driver/executor as well in addition to use of Credentials - and these are driven by hadoop security DT design. > Do we need to generalize ServiceCredentialProvider to support non-hadoop delegation tokens? IMO that depends on what the answer to the design choice above is. If (a) or (c) - then no. If (b), then yes.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org