Maciej Smole��ski created YARN-11397:
----------------------------------------
Summary: Memory leak when reading aggregated logs from s3
(LogAggregationTFileController::readAggregatedLogs)
Key: YARN-11397
URL: https://issues.apache.org/jira/browse/YARN-11397
Project: Hadoop YARN
Issue Type: Bug
Components: log-aggregation
Affects Versions: 3.2.2
Environment: Remote logs dir on s3.
Reporter: Maciej Smole��ski
Attachments: YarnLogsS3Issue.scala
Reproduction code in the attachment.
When collecting aggregated logs from s3 in a loop (see reproduction code) we
can easily see that the number of 'S3AInstrumentation' is increasing although
the number of 'S3AFileSystem' is not increasing. It means that
'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should
be. The root cause of this seems to be the missing close on S3AFileSystem.
Related issues:
https://issues.apache.org/jira/browse/YARN-11039
https://issues.apache.org/jira/browse/HADOOP-18476
The scenario of this bug is different ('thread leak' vs 'memory leak') and
'affected version' are different (3.2.2 in our case), however the root cause
might be the same.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
