[jira] [Commented] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456185#comment-16456185 ] Steve Loughran commented on HBASE-20433: Maybe we could do a special build of hadop-aws which can be set to print a stack trace on creation if debug is set. Or actually do this in in FileSystem for broader debugging of all FS leaks. I Can see the value in that from time to time > HBase Export Snapshot utility does not close FileSystem instances > - > > Key: HBASE-20433 > URL: https://issues.apache.org/jira/browse/HBASE-20433 > Project: HBase > Issue Type: Bug > Components: Client, fs, snapshots >Affects Versions: 1.2.6, 1.4.3 >Reporter: Voyta >Priority: Major > > It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem > instance caching. > When verifySnapshot method is being run it calls often methods like > org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem > but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour > allows allocation of unwanted objects potentially causing memory leaks. > Related issue: https://issues.apache.org/jira/browse/HADOOP-15392 > > Expectation: > * HBase should properly release/close all objects, especially FileSystem > instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454717#comment-16454717 ] Sean Mackrory commented on HBASE-20433: --- I debugged a run of ExportSnapshot in connection with HADOOP-15392. In my case I see an FS instance get created for the snapshot directory (e.g. s3a://my-bucket/hbase-snapshots) and another FS instance get created for the specific snapshot (e.g. s3a://my-bucket/hbase-snapshots/snapshot-name-8015e164-781d-4187-afa1-bcc25541ba9e). The latter is the one where the FS instance is turned off. They also appear to get closed and the metrics system stopped when second one is closed too. There are ~100 files in the snapshot and I just got the 2 FS instances you'd expect, so I don't see how a 53k-file snapshot is resulting in 53k FS instances in the driver. > HBase Export Snapshot utility does not close FileSystem instances > - > > Key: HBASE-20433 > URL: https://issues.apache.org/jira/browse/HBASE-20433 > Project: HBase > Issue Type: Bug > Components: Client, fs, snapshots >Affects Versions: 1.2.6, 1.4.3 >Reporter: Voyta >Priority: Major > > It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem > instance caching. > When verifySnapshot method is being run it calls often methods like > org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem > but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour > allows allocation of unwanted objects potentially causing memory leaks. > Related issue: https://issues.apache.org/jira/browse/HADOOP-15392 > > Expectation: > * HBase should properly release/close all objects, especially FileSystem > instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[
https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441103#comment-16441103
]
Wei-Chiu Chuang commented on HBASE-20433:
-
The other way would be to enable cache again, but at the end (in
{{ExportSnapshot#cleanup()}}), call {{FileSystem#closeAll()}} to purge the
cache.
> HBase Export Snapshot utility does not close FileSystem instances
> -
>
> Key: HBASE-20433
> URL: https://issues.apache.org/jira/browse/HBASE-20433
> Project: HBase
> Issue Type: Bug
> Components: Client, fs, snapshots
>Affects Versions: 1.2.6, 1.4.3
>Reporter: Voyta
>Priority: Major
>
> It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem
> instance caching.
> When verifySnapshot method is being run it calls often methods like
> org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem
> but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour
> allows allocation of unwanted objects potentially causing memory leaks.
> Related issue: https://issues.apache.org/jira/browse/HADOOP-15392
>
> Expectation:
> * HBase should properly release/close all objects, especially FileSystem
> instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441075#comment-16441075 ] Voyta commented on HBASE-20433: --- Hi, I have a heap dump but I cannot share it as it contains sensitive information. So far it seems S3AFileSystem objects are removed from heap but close() method is not called which causes accumulation of metrics as described in https://issues.apache.org/jira/browse/HADOOP-15392 One possibility would be to add finalize() method but it might a bit dirty hack. > HBase Export Snapshot utility does not close FileSystem instances > - > > Key: HBASE-20433 > URL: https://issues.apache.org/jira/browse/HBASE-20433 > Project: HBase > Issue Type: Bug > Components: Client, fs, snapshots >Affects Versions: 1.2.6, 1.4.3 >Reporter: Voyta >Priority: Major > > It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem > instance caching. > When verifySnapshot method is being run it calls often methods like > org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem > but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour > allows allocation of unwanted objects potentially causing memory leaks. > Related issue: https://issues.apache.org/jira/browse/HADOOP-15392 > > Expectation: > * HBase should properly release/close all objects, especially FileSystem > instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441029#comment-16441029 ] Wei-Chiu Chuang commented on HBASE-20433: - Hi Voyta, The reason for not caching FileSystem object is explained in HADOOP-13971. The internal design causes memory leak and has resulted in quite a few tickets in other projects. Yes, therefore the responsibility falls on to the application (like HBase) to manage the FileSystem object & close when appropriate. > HBase Export Snapshot utility does not close FileSystem instances > - > > Key: HBASE-20433 > URL: https://issues.apache.org/jira/browse/HBASE-20433 > Project: HBase > Issue Type: Bug > Components: Client, fs, snapshots >Affects Versions: 1.2.6, 1.4.3 >Reporter: Voyta >Priority: Major > > It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem > instance caching. > When verifySnapshot method is being run it calls often methods like > org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem > but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour > allows allocation of unwanted objects potentially causing memory leaks. > Related issue: https://issues.apache.org/jira/browse/HADOOP-15392 > > Expectation: > * HBase should properly release/close all objects, especially FileSystem > instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
