[ https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved YARN-11039. ----------------------------------- Fix Version/s: 3.4.0 3.3.5 Resolution: Fixed merged in the hadoop patch HADOOP-18476 which fixes this > LogAggregationFileControllerFactory::getFileControllerForRead can leak > threads > ------------------------------------------------------------------------------- > > Key: YARN-11039 > URL: https://issues.apache.org/jira/browse/YARN-11039 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation > Affects Versions: 3.3.4 > Reporter: Rajesh Balamohan > Assignee: Steve Loughran > Priority: Blocker > Labels: performance, stability > Fix For: 3.4.0, 3.3.5 > > > getFileControllerForRead::getFileControllerForRead internally opens up a new > FS object everytime and is not closed. > When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking > KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM. > Lines of interest: > [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167] > {noformat} > try { > Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId, > appOwner); > if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId, > appOwner).hasNext()) { > return fileController; > } > } catch (Exception ex) { > diagnosticsMsg.append(ex.getMessage() + "\n"); > continue; > } > {noformat} > [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252] > {noformat} > > public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf, > Path remoteAppLogDir, ApplicationId appId, String appOwner) > throws IOException { > Path qualifiedLogDir = > FileContext.getFileContext(conf).makeQualified(remoteAppLogDir); > return FileContext.getFileContext( > qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir); > } > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org