[
https://issues.apache.org/jira/browse/HADOOP-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18091:
------------------------------------
Description:
{{ActiveAuditManagerS3A}} uses thread locals to map to active audit spans,
which (because they are wrapped) include back reference to the audit manager
instance and the config it was created with.
these *do not* get cleaned up when the FS instance is closed.
if you have a long lived process creating and destroying many FS instances,
then memory gets used up.
This fix moves off threadlocal into a map of weak references. while a strong
reference is held` (for example in the s3a entry point method) then the
references will always resolve. but if those are released then when a GC is
triggered these weak references will not be retained, so not use up memory
other than entries in the the ha!sh map. the map is held by the s3a auditing
integration, so when the fs is closed, everything is freed up.
was:
{{ActiveAuditManagerS3A}} uses thread locals to map to active audit spans,
which (because they are wrapped) include back reference to the audit manager
instance and the config it was created with.
these *do not* get cleaned up when the FS instance is closed.
if you have a long lived process creating and destroying many FS instances,
then memory gets used up. l
> S3A auditing leaks memory through ThreadLocal references
> --------------------------------------------------------
>
> Key: HADOOP-18091
> URL: https://issues.apache.org/jira/browse/HADOOP-18091
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.3
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> {{ActiveAuditManagerS3A}} uses thread locals to map to active audit spans,
> which (because they are wrapped) include back reference to the audit manager
> instance and the config it was created with.
> these *do not* get cleaned up when the FS instance is closed.
> if you have a long lived process creating and destroying many FS instances,
> then memory gets used up.
> This fix moves off threadlocal into a map of weak references. while a strong
> reference is held` (for example in the s3a entry point method) then the
> references will always resolve. but if those are released then when a GC is
> triggered these weak references will not be retained, so not use up memory
> other than entries in the the ha!sh map. the map is held by the s3a auditing
> integration, so when the fs is closed, everything is freed up.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
