Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/607#issuecomment-41882183
I would think the right thing would be to enclose the entire executor in a
doAs (or save the UGI and keep using it, as this patch is doing, if the former
isn't straightforward). Hadoop UserGroupInformations rest on Java
LoginContexts, which are propagated to child threads when they start.
@tgravescs correct me if I'm wrong, but Hadoop currently has no mechanism
that replaces old delegation tokens with new ones. Even if we do add such a
mechanism, I don't think it would make sense to send credentials with each
task. So I can't see any situation where constructing a UGI for each task
would make sense.
All that said, I remember when dealing with a JT memory leak that there
were some tricky ways that UGIs interact with the FileSystem cache. I'm going
to review the discussion on MAPREDUCE-5508 to try to understand the specifics.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
