[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900007#comment-16900007
]
Gabor Bota edited comment on HADOOP-15565 at 8/5/19 11:21 AM:
--------------------------------------------------------------
Thanks for working on this [~LiJinglun]!
{quote}I think we should let FileSystem.CACHE cache ViewFileSystem only, and
let ViewFileSystem cache all it's child filesystems.{quote}
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of
{{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances
right? If you mean that (based on the code you do) please correct the
description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with {quote}re-login my kerberos
and renew ViewFileSystem periodically{quote}. Could you write a test case where
this is reproduced to show that this change will solve that issue?
was (Author: gabor.bota):
Thanks for working on this [~LiJinglun]!
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let
> ViewFileSystem cache all it's child filesystems.
You mean that the {{FileSystem.CACHE}} will only cache *one instance* of
{{ViewFileSystem}}, and the {{ViewFileSystem}} will handle other instances
right? If you mean that (based on the code you do) please correct the
description.
It would be also good to see more test fore this change:
* Unit tests for the {{ViewFileSystem#SimpleCache}}.
* Based on the description you had a problem with "re-login my kerberos and
renew ViewFileSystem periodically". Could you write a test case where this is
reproduced to show that this change will solve that issue?
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch
>
>
> When we create a ViewFileSystem, all it's child filesystems will be cached by
> FileSystem.CACHE. Unless we close these child filesystems, they will stay in
> FileSystem.CACHE forever.
> I think we should let FileSystem.CACHE cache ViewFileSystem only, and let
> ViewFileSystem cache all it's child filesystems. So we can close
> ViewFileSystem without leak and won't affect other ViewFileSystems.
> I find this problem because i need to re-login my kerberos and renew
> ViewFileSystem periodically. Because FileSystem.CACHE.Key is based on
> UserGroupInformation, which changes everytime i re-login, I can't use the
> cached child filesystems when i new a ViewFileSystem. And because
> ViewFileSystem.close does nothing but remove itself from cache, i leak all
> it's child filesystems in cache.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
