zhihai xu created YARN-3336:

             Summary: FileSystem memory leak in DelegationTokenRenewer
                 Key: YARN-3336
                 URL: https://issues.apache.org/jira/browse/YARN-3336
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
            Reporter: zhihai xu
            Assignee: zhihai xu
            Priority: Critical

FileSystem memory leak in DelegationTokenRenewer.
Every time DelegationTokenRenewer#obtainSystemTokensForUser is called, a new 
FileSystem entry will be added to  FileSystem#CACHE which will never be garbage 
This is the implementation of obtainSystemTokensForUser:
  protected Token<?>[] obtainSystemTokensForUser(String user,
      final Credentials credentials) throws IOException, InterruptedException {
    // Get new hdfs tokens on behalf of this user
    UserGroupInformation proxyUser =
    Token<?>[] newTokens =
        proxyUser.doAs(new PrivilegedExceptionAction<Token<?>[]>() {
          public Token<?>[] run() throws Exception {
            return FileSystem.get(getConfig()).addDelegationTokens(
              UserGroupInformation.getLoginUser().getUserName(), credentials);
    return newTokens;

The memory leak happened when FileSystem.get(getConfig()) is called with a new 
proxy user.
Because createProxyUser will always create a new Subject.
public static UserGroupInformation createProxyUser(String user,
      UserGroupInformation realUser) {
    if (user == null || user.isEmpty()) {
      throw new IllegalArgumentException("Null user");
    if (realUser == null) {
      throw new IllegalArgumentException("Null real user");
    Subject subject = new Subject();
    Set<Principal> principals = subject.getPrincipals();
    principals.add(new User(user));
    principals.add(new RealUser(realUser));
    UserGroupInformation result =new UserGroupInformation(subject);
    return result;

FileSystem#Cache#Key.equals will compare the ugi
      Key(URI uri, Configuration conf, long unique) throws IOException {
        scheme = uri.getScheme()==null?"":uri.getScheme().toLowerCase();
        authority = 
        this.unique = unique;
        this.ugi = UserGroupInformation.getCurrentUser();
      public boolean equals(Object obj) {
        if (obj == this) {
          return true;
        if (obj != null && obj instanceof Key) {
          Key that = (Key)obj;
          return isEqual(this.scheme, that.scheme)
                 && isEqual(this.authority, that.authority)
                 && isEqual(this.ugi, that.ugi)
                 && (this.unique == that.unique);
        return false;        

UserGroupInformation.equals will compare subject by reference.
  public boolean equals(Object o) {
    if (o == this) {
      return true;
    } else if (o == null || getClass() != o.getClass()) {
      return false;
    } else {
      return subject == ((UserGroupInformation) o).subject;

So in this case, every time createProxyUser and FileSystem.get(getConfig()) are 
called, a new FileSystem will be created and a new entry will be added to 

This message was sent by Atlassian JIRA

Reply via email to