[
https://issues.apache.org/jira/browse/YARN-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maciej Smole��ski updated YARN-11397:
-------------------------------------
Description:
Reproduction code in the attachment.
When collecting aggregated logs from s3 in a loop (see reproduction code) we
can easily see that the number of 'S3AInstrumentation' is increasing although
the number of 'S3AFileSystem' is not increasing. It means that
'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should
be. The root cause of this seems to be the missing close on S3AFileSystem.
The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039 but
the issue is a 'memory leak' (not a 'thread leak') and affected version is
earlier here (3.2.2).
was:
Reproduction code in the attachment.
When collecting aggregated logs from s3 in a loop (see reproduction code) we
can easily see that the number of 'S3AInstrumentation' is increasing although
the number of 'S3AFileSystem' is not increasing. It means that
'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should
be. The root cause of this seems to be the missing close on S3AFileSystem.
The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039 but
the issue is a 'memory leak' (not a 'thread leak') and affected version is
earlier here (3.2.2 at least).
> Memory leak when reading aggregated logs from s3
> (LogAggregationTFileController::readAggregatedLogs)
> ----------------------------------------------------------------------------------------------------
>
> Key: YARN-11397
> URL: https://issues.apache.org/jira/browse/YARN-11397
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 3.2.2
> Environment: Remote logs dir on s3.
> Reporter: Maciej Smole��ski
> Priority: Critical
> Attachments: YarnLogsS3Issue.scala
>
>
> Reproduction code in the attachment.
> When collecting aggregated logs from s3 in a loop (see reproduction code) we
> can easily see that the number of 'S3AInstrumentation' is increasing although
> the number of 'S3AFileSystem' is not increasing. It means that
> 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it
> should be. The root cause of this seems to be the missing close on
> S3AFileSystem.
> The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039
> but the issue is a 'memory leak' (not a 'thread leak') and affected version
> is earlier here (3.2.2).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
