[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104861#comment-17104861 ] Virajith Jalaparti edited comment on HADOOP-15565 at 5/11/20, 7:56 PM: --- Thanks [~umamaheswararao]. The test failures in the last jenkins run are not related here; the javac issues are warnings – will leave them to remain similar to the original patch. Will commit [^HADOOP-15565-branch-3.2.002.patch] soon. was (Author: virajith): Thanks [~umamaheswararao]. The test failures in the last jenkins run are not related here; the javac issues are warnings – changing these will not be a simple backport. Will commit [^HADOOP-15565-branch-3.2.002.patch] soon. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923545#comment-16923545
]
Erik Krogen edited comment on HADOOP-15565 at 9/5/19 3:41 PM:
--
Thanks for the detailed explanations [~LiJinglun]! They are very helpful; it
all makes sense now.
It looks like the v007 patch is identical to the v006 patch – did you upload
the wrong file?
{code:java}
± diff HADOOP-15565.0006.patch HADOOP-15565.0007.patch | wc -l
0
{code}
While I'm at it, there is one more thing I noticed: In
{{TestChRootedFileSystem#getChildFileSystem()}}, can we use
[{{Objects.equals}}|https://docs.oracle.com/javase/8/docs/api/java/util/Objects.html#equals-java.lang.Object-java.lang.Object-]
instead of manually doing a null + equality check? I think it should make this
cleaner.
was (Author: xkrogen):
Thanks for the detailed explanations [~LiJinglun]! They are very helpful; it
all makes sense now.
It looks like the v007 patch is identical to the v006 patch – did you upload
the wrong file?
{code:java}
± diff HADOOP-15565.0006.patch HADOOP-15565.0006.patch | wc -l
0
{code}
While I'm at it, there is one more thing I noticed: In
{{TestChRootedFileSystem#getChildFileSystem()}}, can we use
[{{Objects.equals}}|https://docs.oracle.com/javase/8/docs/api/java/util/Objects.html#equals-java.lang.Object-java.lang.Object-]
instead of manually doing a null + equality check? I think it should make this
cleaner.
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
>
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch,
> HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch,
> HADOOP-15565.0006.patch, HADOOP-15565.0007.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
> It's children filesystems are cached in FileSystem.CACHE and shared by all
> the ViewFileSystem instances. We could't simply close all the children
> filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children
> filesystems. The children filesystems are not shared any more. When
> ViewFileSystem is closed we close all the children filesystems in the inner
> cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't
> be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other
> instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918429#comment-16918429 ] Jinglun edited comment on HADOOP-15565 at 8/29/19 9:11 AM: --- Hi [~xkrogen] [~jojochuang] , could you help to review this, thanks ! was (Author: lijinglun): Hi [~xkrogen] [~xkrogen], could you help to review this, thanks ! > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, > HADOOP-15565.0006.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1697#comment-1697
]
Gabor Bota edited comment on HADOOP-15565 at 8/5/19 11:21 AM:
--
Thanks for working on this [~LiJinglun]!
{quote}I think we should let FileSystem.CACHE cache ViewFileSystem only, and
let ViewFileSystem cache all it's child filesystems.{quote}
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of
{{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances
right? If you mean that (based on the code you do) please correct the
description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with {quote}re-login my kerberos
and renew ViewFileSystem periodically{quote}. Could you write a test case where
this is reproduced to show that this change will solve that issue?
was (Author: gabor.bota):
Thanks for working on this [~LiJinglun]!
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let
> ViewFileSystem cache all it's child filesystems.
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of
{{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances
right? If you mean that (based on the code you do) please correct the
description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with "re-login my kerberos and
renew ViewFileSystem periodically". Could you write a test case where this is
reproduced to show that this change will solve that issue?
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
>
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch
>
>
> When we create a ViewFileSystem, all it's child filesystems will be cached by
> FileSystem.CACHE. Unless we close these child filesystems, they will stay in
> FileSystem.CACHE forever.
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let
> ViewFileSystem cache all it's child filesystems. So we can close
> ViewFileSystem without leak and won't affect other ViewFileSystems.
> I find this problem because i need to re-login my kerberos and renew
> ViewFileSystem periodically. Because FileSystem.CACHE.Key is based on
> UserGroupInformation, which changes everytime i re-login, I can't use the
> cached child filesystems when i new a ViewFileSystem. And because
> ViewFileSystem.close does nothing but remove itself from cache, i leak all
> it's child filesystems in cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
