[
https://issues.apache.org/jira/browse/YARN-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved YARN-11039.
-----------------------------------
Fix Version/s: 3.4.0
3.3.5
Resolution: Fixed
merged in the hadoop patch HADOOP-18476 which fixes this
> LogAggregationFileControllerFactory::getFileControllerForRead can leak
> threads
> -------------------------------------------------------------------------------
>
> Key: YARN-11039
> URL: https://issues.apache.org/jira/browse/YARN-11039
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 3.3.4
> Reporter: Rajesh Balamohan
> Assignee: Steve Loughran
> Priority: Blocker
> Labels: performance, stability
> Fix For: 3.4.0, 3.3.5
>
>
> getFileControllerForRead::getFileControllerForRead internally opens up a new
> FS object everytime and is not closed.
> When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking
> KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.
> Lines of interest:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167]
> {noformat}
> try {
> Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
> appOwner);
> if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
> appOwner).hasNext()) {
> return fileController;
> }
> } catch (Exception ex) {
> diagnosticsMsg.append(ex.getMessage() + "\n");
> continue;
> }
> {noformat}
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252]
> {noformat}
>
> public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
> Path remoteAppLogDir, ApplicationId appId, String appOwner)
> throws IOException {
> Path qualifiedLogDir =
> FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
> return FileContext.getFileContext(
> qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
