[jira] [Resolved] (HADOOP-18515) Backport HADOOP-17612 to branch-3.3
[ https://issues.apache.org/jira/browse/HADOOP-18515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti resolved HADOOP-18515. - Resolution: Fixed > Backport HADOOP-17612 to branch-3.3 > --- > > Key: HADOOP-18515 > URL: https://issues.apache.org/jira/browse/HADOOP-18515 > Project: Hadoop Common > Issue Type: Improvement > Components: auth, common, nfs, registry >Affects Versions: 3.3.5 >Reporter: Melissa You >Assignee: Melissa You >Priority: Major > Labels: pull-request-available > > This is a sub-task of HADOOP-18518 to upgrade zk on 3.3 branches. > It is a clean cherry pick from [https://github.com/apache/hadoop/pull/3241] . -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18193) Support nested mount points in INodeTree
[
https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532517#comment-17532517
]
Virajith Jalaparti commented on HADOOP-18193:
-
Thanks for the doc [~Lei Yang]. A few comments:
# Nit: Nested paths and Nested mount points seem to be used interchangeably -
Can you disambiguate them in the doc and explicitly define what they mean?
# It's unclear what DMT is in the requirements section - can you keep the doc
specific to this JIRA?
# The high-level approach section is very useful, thanks for writing it up!
> Support nested mount points in INodeTree
>
>
> Key: HADOOP-18193
> URL: https://issues.apache.org/jira/browse/HADOOP-18193
> Project: Hadoop Common
> Issue Type: Improvement
> Components: viewfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Assignee: Lei Yang
>Priority: Major
> Labels: pull-request-available
> Attachments: Nested Mount Point in ViewFs.pdf
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Defining following client mount table config is not supported in INodeTree
> and will throw FileAlreadyExistsException
>
> {code:java}
> fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar
> fs.viewfs.mounttable.link./foo=hdfs://nn02/foo
> {code}
> INodeTree has 2 methods that need change to support nested mount points.
> {code:java}
> createLink(): build INodeTree during fs init.
> resolve(): resolve path in INodeTree with viewfs apis.
> {code}
> ViewFileSystem and ViewFs maintains an INodeTree instance(fsState) in both
> classes and call fsState.resolve(..) to resolve path to specific mount point.
> INodeTree.resolve encapsulates the logic of nested mount point resolving. So
> no changes are expected in both classes.
> AC:
> # INodeTree.createlink should support creating nested mount
> points.(INodeTree is constructed during fs init)
> # INodeTree.resolve should support resolve path based on nested mount
> points. (INodeTree.resolve is used in viewfs apis)
> # No regression in existing ViewFileSystem and ViewFs apis.
> # Ensure some important apis are not broken with nested mount points.
> (Rename, getContentSummary, listStatus...)
>
> Spec:
> Please review attached pdf for spec about this feature.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18193) Support nested mount points in INodeTree
[ https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531329#comment-17531329 ] Virajith Jalaparti commented on HADOOP-18193: - Ping on this [~umamaheswararao], [~ayushtkn], [~vinayakumarb]. > Support nested mount points in INodeTree > > > Key: HADOOP-18193 > URL: https://issues.apache.org/jira/browse/HADOOP-18193 > Project: Hadoop Common > Issue Type: Improvement > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Attachments: Nested Mount Point in ViewFs.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Defining following client mount table config is not supported in INodeTree > and will throw FileAlreadyExistsException > fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar > fs.viewfs.mounttable.link./foo=hdfs://nn02/foo > > INodeTree has 2 methods that need change to support nested mount points. > createLink(..): build INodeTree during fs init. > resolve(..): resolve path in INodeTree with viewfs apis. > > ViewFileSystem and ViewFs referes INodeTree.resolve(..) to resolve path to > specific mount point. No changes are expected in both classes. However, we > need to support existing use cases and make sure no regression are caused. > > AC: > # INodeTree.createlink should support creating nested mount > points.(INodeTree is constructed during fs init) > # INodeTree.resolve should support resolve path based on nested mount > points. (INodeTree.resolve is used in viewfs apis) > # No regression in existing ViewFileSystem and ViewFs apis. > # Ensure some important apis are not broken with nested mount points. > (Rename, getContentSummary, listStatus...) -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18172) Change scope of getRootFallbackLink for InodeTree to make them accessible from outside package
[ https://issues.apache.org/jira/browse/HADOOP-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-18172: Fix Version/s: 3.4.0 3.3.4 > Change scope of getRootFallbackLink for InodeTree to make them accessible > from outside package > -- > > Key: HADOOP-18172 > URL: https://issues.apache.org/jira/browse/HADOOP-18172 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Sometimes, we need to access rootFallBackLink in InodeTree from another > package. One such case is we extend from ViewFileSystem but want to put the > new filesystem in org.apache.hadoop.fs package, instead of > org.apache.hadoop.fs.viewfs package. As a result, we need make them public, > similar as what we did for getMountPoints() in HADOOP-18100. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18193) Support nested mount points in INodeTree
[ https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17523913#comment-17523913 ] Virajith Jalaparti commented on HADOOP-18193: - Hi [~umamaheswararao], [~ayushtkn], [~vinayakumarb] can one of you help review this PR? > Support nested mount points in INodeTree > > > Key: HADOOP-18193 > URL: https://issues.apache.org/jira/browse/HADOOP-18193 > Project: Hadoop Common > Issue Type: Improvement > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Defining following client mount table config is not supported in INodeTree > and will throw FileAlreadyExistsException > fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar > fs.viewfs.mounttable.link./foo=hdfs://nn02/foo > > INodeTree has 2 methods that need change to support nested mount points. > createLink(..): build INodeTree during fs init. > resolve(..): resolve path in INodeTree with viewfs apis. > > ViewFileSystem and ViewFs referes INodeTree.resolve(..) to resolve path to > specific mount point. No changes are expected in both classes. However, we > need to support existing use cases and make sure no regression are caused. > > AC: > # INodeTree.createlink should support creating nested mount > points.(INodeTree is constructed during fs init) > # INodeTree.resolve should support resolve path based on nested mount > points. (INodeTree.resolve is used in viewfs apis) > # No regression in existing ViewFileSystem and ViewFs apis. > # Ensure some important apis are not broken with nested mount points. > (Rename, getContentSummary, listStatus...) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18193) Support nested mount points in INodeTree
[ https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HADOOP-18193: --- Assignee: Lei Yang (was: Virajith Jalaparti) > Support nested mount points in INodeTree > > > Key: HADOOP-18193 > URL: https://issues.apache.org/jira/browse/HADOOP-18193 > Project: Hadoop Common > Issue Type: Improvement > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Defining following client mount table config is not supported in INodeTree > and will throw FileAlreadyExistsException > fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar > fs.viewfs.mounttable.link./foo=hdfs://nn02/foo > > INodeTree has 2 methods that need change to support nested mount points. > createLink(..): build INodeTree during fs init. > resolve(..): resolve path in INodeTree with viewfs apis. > > ViewFileSystem and ViewFs referes INodeTree.resolve(..) to resolve path to > specific mount point. No changes are expected in both classes. However, we > need to support existing use cases and make sure no regression are caused. > > AC: > # INodeTree.createlink should support creating nested mount > points.(INodeTree is constructed during fs init) > # INodeTree.resolve should support resolve path based on nested mount > points. (INodeTree.resolve is used in viewfs apis) > # No regression in existing ViewFileSystem and ViewFs apis. > # Ensure some important apis are not broken with nested mount points. > (Rename, getContentSummary, listStatus...) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18193) Support nested mount points in INodeTree
[ https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HADOOP-18193: --- Assignee: Virajith Jalaparti > Support nested mount points in INodeTree > > > Key: HADOOP-18193 > URL: https://issues.apache.org/jira/browse/HADOOP-18193 > Project: Hadoop Common > Issue Type: Improvement > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Lei Yang >Assignee: Virajith Jalaparti >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Defining following client mount table config is not supported in INodeTree > and will throw FileAlreadyExistsException > fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar > fs.viewfs.mounttable.link./foo=hdfs://nn02/foo > > INodeTree has 2 methods that need change to support nested mount points. > createLink(..): build INodeTree during fs init. > resolve(..): resolve path in INodeTree with viewfs apis. > > ViewFileSystem and ViewFs referes INodeTree.resolve(..) to resolve path to > specific mount point. No changes are expected in both classes. However, we > need to support existing use cases and make sure no regression are caused. > > AC: > # INodeTree.createlink should support creating nested mount > points.(INodeTree is constructed during fs init) > # INodeTree.resolve should support resolve path based on nested mount > points. (INodeTree.resolve is used in viewfs apis) > # No regression in existing ViewFileSystem and ViewFs apis. > # Ensure some important apis are not broken with nested mount points. > (Rename, getContentSummary, listStatus...) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1712#comment-1712
]
Virajith Jalaparti commented on HADOOP-17072:
-
Thanks for the feedback! [~ste...@apache.org] , thanks for listing the
requirements for FileSystem changes.
I am inclined to agree with [~umamaheswararao] 's suggestion of having this in
a util class and not in FileSystem as sufficient APIs are already exposed to
enable the same functionality. At this point, any application can actually do
this implementation themselves and there's not much to do at the FS layer.
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HADOOP-17072.001.patch
>
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider the example of GDPR compliance/retention jobs that need to go over
> various datasets, ingested over a period of T days and remove/quarantine
> datasets that are not properly annotated/have reached their retention period.
> Such jobs can rely on renames to a global trash/quarantine directory to
> accomplish their task. However, in a federated setting, efficient, atomic
> renames (as those within a single HDFS cluster) are not supported across the
> different clusters/shards in federation. As a result, such jobs will need to
> leverage a trash/quarantine directory per cluster/shard. Further, they would
> need to map from a particular path to the cluster/shard that contains this
> path.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17139505#comment-17139505
]
Virajith Jalaparti commented on HADOOP-17072:
-
[~umamahesh], [~stevel], [~arpaga], [~weichiu] as you have been looking at
ViewFS recently. Any thoughts on this?
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HADOOP-17072.001.patch
>
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider the example of GDPR compliance/retention jobs that need to go over
> various datasets, ingested over a period of T days and remove/quarantine
> datasets that are not properly annotated/have reached their retention period.
> Such jobs can rely on renames to a global trash/quarantine directory to
> accomplish their task. However, in a federated setting, efficient, atomic
> renames (as those within a single HDFS cluster) are not supported across the
> different clusters/shards in federation. As a result, such jobs will need to
> leverage a trash/quarantine directory per cluster/shard. Further, they would
> need to map from a particular path to the cluster/shard that contains this
> path.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138049#comment-17138049
]
Virajith Jalaparti commented on HADOOP-17072:
-
Adding [^HADOOP-17072.001.patch] to show how this will be implemented.
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HADOOP-17072.001.patch
>
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider the example of GDPR compliance/retention jobs that need to go over
> various datasets, ingested over a period of T days and remove/quarantine
> datasets that are not properly annotated/have reached their retention period.
> Such jobs can rely on renames to a global trash/quarantine directory to
> accomplish their task. However, in a federated setting, efficient, atomic
> renames (as those within a single HDFS cluster) are not supported across the
> different clusters/shards in federation. As a result, such jobs will need to
> leverage a trash/quarantine directory per cluster/shard. Further, they would
> need to map from a particular path to the cluster/shard that contains this
> path.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HADOOP-17072:
Description:
In a federated setting (HDFS federation, federation across multiple buckets on
S3, multiple containers across Azure storage), certain system tools/pipelines
require the ability to map paths to the clusters/accounts.
Consider the example of GDPR compliance/retention jobs that need to go over
various datasets, ingested over a period of T days and remove/quarantine
datasets that are not properly annotated/have reached their retention period.
Such jobs can rely on renames to a global trash/quarantine directory to
accomplish their task. However, in a federated setting, efficient, atomic
renames (as those within a single HDFS cluster) are not supported across the
different clusters/shards in federation. As a result, such jobs will need to
leverage a trash/quarantine directory per cluster/shard. Further, they would
need to map from a particular path to the cluster/shard that contains this path.
To address such cases, this JIRA proposes to get add two new methods to
{{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
was:
In a federated setting (HDFS federation, federation across multiple buckets on
S3, multiple containers across Azure storage), certain system tools/pipelines
require the ability to map paths to the clusters/accounts.
Consider GDPR compliance/retention jobs need to go over the datasets ingested
over a period of T days and remove/quarantine datasets that are not properly
annotated/have reached their retention period. Such jobs can rely on renames to
a global trash/quarantine directory to accomplish their task. However, in a
federated setting, efficient, atomic renames (as those within a single HDFS
cluster) are not supported across the different clusters/shards in federation.
As a result, such jobs will need to get the clusters to which different paths
map to.
To address such cases, this JIRA proposes to get add two new methods to
{{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider the example of GDPR compliance/retention jobs that need to go over
> various datasets, ingested over a period of T days and remove/quarantine
> datasets that are not properly annotated/have reached their retention period.
> Such jobs can rely on renames to a global trash/quarantine directory to
> accomplish their task. However, in a federated setting, efficient, atomic
> renames (as those within a single HDFS cluster) are not supported across the
> different clusters/shards in federation. As a result, such jobs will need to
> leverage a trash/quarantine directory per cluster/shard. Further, they would
> need to map from a particular path to the cluster/shard that contains this
> path.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti reassigned HADOOP-17072:
---
Assignee: Virajith Jalaparti
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider GDPR compliance/retention jobs need to go over the datasets ingested
> over a period of T days and remove/quarantine datasets that are not properly
> annotated/have reached their retention period. Such jobs can rely on renames
> to a global trash/quarantine directory to accomplish their task. However, in
> a federated setting, efficient, atomic renames (as those within a single HDFS
> cluster) are not supported across the different clusters/shards in
> federation. As a result, such jobs will need to get the clusters to which
> different paths map to.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HADOOP-17072:
Attachment: HADOOP-17072.001.patch
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HADOOP-17072.001.patch
>
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider the example of GDPR compliance/retention jobs that need to go over
> various datasets, ingested over a period of T days and remove/quarantine
> datasets that are not properly annotated/have reached their retention period.
> Such jobs can rely on renames to a global trash/quarantine directory to
> accomplish their task. However, in a federated setting, efficient, atomic
> renames (as those within a single HDFS cluster) are not supported across the
> different clusters/shards in federation. As a result, such jobs will need to
> leverage a trash/quarantine directory per cluster/shard. Further, they would
> need to map from a particular path to the cluster/shard that contains this
> path.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
[
https://issues.apache.org/jira/browse/HADOOP-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HADOOP-17072:
Description:
In a federated setting (HDFS federation, federation across multiple buckets on
S3, multiple containers across Azure storage), certain system tools/pipelines
require the ability to map paths to the clusters/accounts.
Consider GDPR compliance/retention jobs need to go over the datasets ingested
over a period of T days and remove/quarantine datasets that are not properly
annotated/have reached their retention period. Such jobs can rely on renames to
a global trash/quarantine directory to accomplish their task. However, in a
federated setting, efficient, atomic renames (as those within a single HDFS
cluster) are not supported across the different clusters/shards in federation.
As a result, such jobs will need to get the clusters to which different paths
map to.
To address such cases, this JIRA proposes to get add two new methods to
{{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
was:
In a federated setting (HDFS federation, federation across multiple buckets on
S3, multiple containers across Azure storage), certain system tools/pipelines
require the ability to map paths to the clusters/accounts.
Consider GDPR compliance/retention jobs need to go over the datasets ingested
over a period of T days and remove/quarantine datasets that are not properly
annotated/have reached their retention period. Such jobs can rely on renames to
a global trash/quarantine directory to accomplish their task. However, in a
federated setting, efficient, atomic renames (as those within a single HDFS
cluster) are not supported across the different clusters/shards in federation.
As a result, such jobs will need to get the clusters to which different paths
map to.
To address such cases, this JIRA proposed to get add two new methods to
{{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
> Add getClusterRoot and getClusterRoots methods to FileSystem and
> ViewFilesystem
> ---
>
> Key: HADOOP-17072
> URL: https://issues.apache.org/jira/browse/HADOOP-17072
> Project: Hadoop Common
> Issue Type: Task
> Components: fs, viewfs
>Reporter: Virajith Jalaparti
>Priority: Major
>
> In a federated setting (HDFS federation, federation across multiple buckets
> on S3, multiple containers across Azure storage), certain system
> tools/pipelines require the ability to map paths to the clusters/accounts.
> Consider GDPR compliance/retention jobs need to go over the datasets ingested
> over a period of T days and remove/quarantine datasets that are not properly
> annotated/have reached their retention period. Such jobs can rely on renames
> to a global trash/quarantine directory to accomplish their task. However, in
> a federated setting, efficient, atomic renames (as those within a single HDFS
> cluster) are not supported across the different clusters/shards in
> federation. As a result, such jobs will need to get the clusters to which
> different paths map to.
> To address such cases, this JIRA proposes to get add two new methods to
> {{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17072) Add getClusterRoot and getClusterRoots methods to FileSystem and ViewFilesystem
Virajith Jalaparti created HADOOP-17072:
---
Summary: Add getClusterRoot and getClusterRoots methods to
FileSystem and ViewFilesystem
Key: HADOOP-17072
URL: https://issues.apache.org/jira/browse/HADOOP-17072
Project: Hadoop Common
Issue Type: Task
Components: fs, viewfs
Reporter: Virajith Jalaparti
In a federated setting (HDFS federation, federation across multiple buckets on
S3, multiple containers across Azure storage), certain system tools/pipelines
require the ability to map paths to the clusters/accounts.
Consider GDPR compliance/retention jobs need to go over the datasets ingested
over a period of T days and remove/quarantine datasets that are not properly
annotated/have reached their retention period. Such jobs can rely on renames to
a global trash/quarantine directory to accomplish their task. However, in a
federated setting, efficient, atomic renames (as those within a single HDFS
cluster) are not supported across the different clusters/shards in federation.
As a result, such jobs will need to get the clusters to which different paths
map to.
To address such cases, this JIRA proposed to get add two new methods to
{{FileSystem}}: {{getClusterRoot}} and {{getClusterRoots()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Resolution: Fixed Status: Resolved (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Fix Version/s: 2.10.1 3.1.4 > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105964#comment-17105964 ] Virajith Jalaparti commented on HADOOP-15565: - Committed [^HADOOP-15565-branch-3.1.001.patch] and [^HADOOP-15565-branch-2.10.001.patch] to branches 3.1 and 2.10 resp. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105852#comment-17105852 ] Virajith Jalaparti commented on HADOOP-15565: - Thanks for taking a look [~umamaheswararao]! Will commit these soon. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105617#comment-17105617 ] Virajith Jalaparti commented on HADOOP-15565: - Looks like the issues in [the last jenkins run| https://issues.apache.org/jira/browse/HADOOP-15565?focusedCommentId=17105143=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17105143] are from 2.10 and not related to [^HADOOP-15565-branch-2.10.001.patch] > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105080#comment-17105080 ] Virajith Jalaparti commented on HADOOP-15565: - Hi [~umamaheswararao] - do you mind checking [^HADOOP-15565-branch-3.1.001.patch] and [^HADOOP-15565-branch-2.10.001.patch]? Straight forward but had minor conflicts when I backported. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-2.10.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-3.1.001.patch, > HADOOP-15565-branch-3.2.001.patch, HADOOP-15565-branch-3.2.002.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.1.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-3.1.001.patch, > HADOOP-15565-branch-3.2.001.patch, HADOOP-15565-branch-3.2.002.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-3.1.001.patch, > HADOOP-15565-branch-3.2.001.patch, HADOOP-15565-branch-3.2.002.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Fix Version/s: 3.2.2 > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104861#comment-17104861 ] Virajith Jalaparti edited comment on HADOOP-15565 at 5/11/20, 7:56 PM: --- Thanks [~umamaheswararao]. The test failures in the last jenkins run are not related here; the javac issues are warnings – will leave them to remain similar to the original patch. Will commit [^HADOOP-15565-branch-3.2.002.patch] soon. was (Author: virajith): Thanks [~umamaheswararao]. The test failures in the last jenkins run are not related here; the javac issues are warnings – changing these will not be a simple backport. Will commit [^HADOOP-15565-branch-3.2.002.patch] soon. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104861#comment-17104861 ] Virajith Jalaparti commented on HADOOP-15565: - Thanks [~umamaheswararao]. The test failures in the last jenkins run are not related here; the javac issues are warnings – changing these will not be a simple backport. Will commit [^HADOOP-15565-branch-3.2.002.patch] soon. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104509#comment-17104509
]
Virajith Jalaparti commented on HADOOP-15565:
-
Good catch [~umamaheswararao]. I removed {{removeFileSystemForTesting}} in
[^HADOOP-15565-branch-3.2.002.patch]
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
>
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15565-branch-3.2.001.patch,
> HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch,
> HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch,
> HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch,
> HADOOP-15565.0007.patch, HADOOP-15565.0008.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
> It's children filesystems are cached in FileSystem.CACHE and shared by all
> the ViewFileSystem instances. We could't simply close all the children
> filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children
> filesystems. The children filesystems are not shared any more. When
> ViewFileSystem is closed we close all the children filesystems in the inner
> cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't
> be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other
> instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.2.002.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565-branch-3.2.002.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: (was: HADOOP-15565-branch-3.2.001.patch) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.2.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[
https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103025#comment-17103025
]
Virajith Jalaparti commented on HADOOP-15565:
-
Failed test {{TestRollingUpgrade}} seems to be a transient failure here (passes
locally) - re-queuing jenkins. [~umamahesh]/[~cliang] - do you mind reviewing
[^HADOOP-15565-branch-3.2.001.patch]? Backporting this all the way to 2.10.
> ViewFileSystem.close doesn't close child filesystems and causes FileSystem
> objects leak.
>
>
> Key: HADOOP-15565
> URL: https://issues.apache.org/jira/browse/HADOOP-15565
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15565-branch-3.2.001.patch,
> HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch,
> HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak,
> HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch
>
>
> ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE.
> It's children filesystems are cached in FileSystem.CACHE and shared by all
> the ViewFileSystem instances. We could't simply close all the children
> filesystems because it will break the semantic of FileSystem.newInstance().
> We might add an inner cache to ViewFileSystem, let it cache all the children
> filesystems. The children filesystems are not shared any more. When
> ViewFileSystem is closed we close all the children filesystems in the inner
> cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't
> be too many FileSystem instances.
> The FileSystem.CACHE caches the ViewFileSysem instance and the other
> instances(the children filesystems) are cached in the inner cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.2.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: (was: HADOOP-15565-branch-2.10.001.patch) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: (was: HADOOP-15565-branch-3.1.001.patch) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Open) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: (was: HADOOP-15565-branch-3.2.001.patch) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Open (was: Patch Available) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Status: Patch Available (was: Reopened) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-2.10.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-2.10.001.patch, > HADOOP-15565-branch-3.1.001.patch, HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.1.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.1.001.patch, > HADOOP-15565-branch-3.2.001.patch, HADOOP-15565.0001.patch, > HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, > HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, > HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15565: Attachment: HADOOP-15565-branch-3.2.001.patch > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565-branch-3.2.001.patch, > HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, HADOOP-15565.0003.patch, > HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, HADOOP-15565.0006.bak, > HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HADOOP-15565: --- Assignee: Virajith Jalaparti (was: Jinglun) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, > HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, > HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HADOOP-15565: --- Assignee: Jinglun (was: Virajith Jalaparti) > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, > HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, > HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reopened HADOOP-15565: - Re-opening this issue to backport to 2.10. > ViewFileSystem.close doesn't close child filesystems and causes FileSystem > objects leak. > > > Key: HADOOP-15565 > URL: https://issues.apache.org/jira/browse/HADOOP-15565 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15565.0001.patch, HADOOP-15565.0002.patch, > HADOOP-15565.0003.patch, HADOOP-15565.0004.patch, HADOOP-15565.0005.patch, > HADOOP-15565.0006.bak, HADOOP-15565.0006.patch, HADOOP-15565.0007.patch, > HADOOP-15565.0008.patch > > > ViewFileSystem.close() does nothing but remove itself from FileSystem.CACHE. > It's children filesystems are cached in FileSystem.CACHE and shared by all > the ViewFileSystem instances. We could't simply close all the children > filesystems because it will break the semantic of FileSystem.newInstance(). > We might add an inner cache to ViewFileSystem, let it cache all the children > filesystems. The children filesystems are not shared any more. When > ViewFileSystem is closed we close all the children filesystems in the inner > cache. The ViewFileSystem is still cached by FileSystem.CACHE so there won't > be too many FileSystem instances. > The FileSystem.CACHE caches the ViewFileSysem instance and the other > instances(the children filesystems) are cached in the inner cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17024) ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).
[
https://issues.apache.org/jira/browse/HADOOP-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099162#comment-17099162
]
Virajith Jalaparti commented on HADOOP-17024:
-
We also saw other issues where some aggregation/recursive calls like
{{contentSummary}} may not give the right result when called on Inodes in the
mount table. We wanted to a few enhancements to ViewFS –
[~cliang]/[~abhishekd], can you file JIRAs for these? thanks!
> ListStatus on ViewFS root (ls "/") should list the linkFallBack root
> (configured target root).
> --
>
> Key: HADOOP-17024
> URL: https://issues.apache.org/jira/browse/HADOOP-17024
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, viewfs
>Affects Versions: 3.2.2
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> As part of the design doc HDFS-15289, [~sanjay.radia] and me discussed the
> following scenarios when fallback enabled.
> *Behavior when fallback enabled:*
> Assume FS trees and mount mappings like below:
> mount link /a/b/c/d → hdfs://nn1/a/b
> mount link /a/p/q/r → hdfs://nn2/a/b
> fallback → hdfs://nn3/ $ /a/c
> /x/z
> # Open(/x/y) then it goes to nn3 (fallback) - WORKS
> # Create(/x/foo) then foo is created in nn3 in dir /x - WORKS
> # ls / should list /a /x .Today this does not work and IT IS A BUG!!!
> Because it conflicts with the open(/x/y)
> # Create /y : fails - also fails when not using fallback - WORKS
> # Create /a/z : fails - also fails when not using fallback - WORKS
> # ls /a should list /b /p as expected and will not show fallback in nn3 -
> WORKS
>
> This Jira will fix issue of #3. So, when fallback enabled it should show
> merged ls view with mount links + fallback root. ( this will only be at root
> level)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17024) ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).
[
https://issues.apache.org/jira/browse/HADOOP-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099157#comment-17099157
]
Virajith Jalaparti commented on HADOOP-17024:
-
Hi [~umamaheswararao], yes, we saw this issue. Can you clarify the mount table
in your example? IIUC, I agree that an ls on / here should list whatever is
configured in the {{/linkFallBack}} as well.
cc: [~cliang], [~abhishekd].
> ListStatus on ViewFS root (ls "/") should list the linkFallBack root
> (configured target root).
> --
>
> Key: HADOOP-17024
> URL: https://issues.apache.org/jira/browse/HADOOP-17024
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, viewfs
>Affects Versions: 3.2.2
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> As part of the design doc HDFS-15289, [~sanjay.radia] and me discussed the
> following scenarios when fallback enabled.
> *Behavior when fallback enabled:*
> Assume FS trees and mount mappings like below:
> mount link /a/b/c/d → hdfs://nn1/a/b
> mount link /a/p/q/r → hdfs://nn2/a/b
> fallback → hdfs://nn3/ $ /a/c
> /x/z
> # Open(/x/y) then it goes to nn3 (fallback) - WORKS
> # Create(/x/foo) then foo is created in nn3 in dir /x - WORKS
> # ls / should list /a /x .Today this does not work and IT IS A BUG!!!
> Because it conflicts with the open(/x/y)
> # Create /y : fails - also fails when not using fallback - WORKS
> # Create /a/z : fails - also fails when not using fallback - WORKS
> # ls /a should list /b /p as expected and will not show fallback in nn3 -
> WORKS
>
> This Jira will fix issue of #3. So, when fallback enabled it should show
> merged ls view with mount links + fallback root. ( this will only be at root
> level)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16762) Add support for Filesystem#getFileChecksum in ABFS driver
[ https://issues.apache.org/jira/browse/HADOOP-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995969#comment-16995969 ] Virajith Jalaparti commented on HADOOP-16762: - Hi [~bilahari.th] and [~snvijaya], do you have any plans to add this API to the ABFS driver? > Add support for Filesystem#getFileChecksum in ABFS driver > - > > Key: HADOOP-16762 > URL: https://issues.apache.org/jira/browse/HADOOP-16762 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > Currently, ABFS driver does not support Filesystem#getFileChecksum even > though the underlying ADLS REST API does. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16762) Add support for Filesystem#getFileChecksum in ABFS driver
Virajith Jalaparti created HADOOP-16762: --- Summary: Add support for Filesystem#getFileChecksum in ABFS driver Key: HADOOP-16762 URL: https://issues.apache.org/jira/browse/HADOOP-16762 Project: Hadoop Common Issue Type: Sub-task Reporter: Virajith Jalaparti Currently, ABFS driver does not support Filesystem#getFileChecksum even though the underlying ADLS REST API does. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16734) Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989258#comment-16989258 ] Virajith Jalaparti commented on HADOOP-16734: - Thanks for posting this [~bilahari.th]. Can you also create a PR for branch-2.10? I can do the cherry-pick but I am not sure if other changes need to go in first, > Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to > branch-2 > --- > > Key: HADOOP-16734 > URL: https://issues.apache.org/jira/browse/HADOOP-16734 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 2.11.0 > > > Backport https://issues.apache.org/jira/browse/HADOOP-16455 to branch-2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16640) WASB: Override getCanonicalServiceName() to return full url of WASB filesystem
[ https://issues.apache.org/jira/browse/HADOOP-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-16640: Fix Version/s: 3.2.2 > WASB: Override getCanonicalServiceName() to return full url of WASB filesystem > -- > > Key: HADOOP-16640 > URL: https://issues.apache.org/jira/browse/HADOOP-16640 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.2 > > > HBase calls getCanonicalServiceName() to check if two FS are the same: > [https://github.com/apache/hbase/blob/10180e232ebf886c9577d77eb91ce64b51564dfc/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L117] > This is creating some issues for customer because the current WASB relied on > the default implementation of getCanonicalServiceName() and will return > "ip:port". > Will override getCanonicalServiceName() in WASB to return the full URI of > the fs, and this would be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983886#comment-16983886 ] Virajith Jalaparti commented on HADOOP-16455: - Thanks for completing this [~bilahari.th]. Do you plan to backport to branch-2 as well? > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.0 > > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972050#comment-16972050 ] Virajith Jalaparti edited comment on HADOOP-16455 at 11/12/19 4:13 AM: --- Hi [~bilahari.th], [~DanielZhou], Any updates/ETA on this? We are interested in having his implemented in ABFS as well and ported back to branch-2.10. was (Author: virajith): Hi [~bilahari.th], [~DanielZhou], Any updates/ETA on this? We would like to have this implemented in ABFS as well. > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16455) ABFS: Implement FileSystem.access() method
[ https://issues.apache.org/jira/browse/HADOOP-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972050#comment-16972050 ] Virajith Jalaparti commented on HADOOP-16455: - Hi [~bilahari.th], [~DanielZhou], Any updates/ETA on this? We would like to have this implemented in ABFS as well. > ABFS: Implement FileSystem.access() method > -- > > Key: HADOOP-16455 > URL: https://issues.apache.org/jira/browse/HADOOP-16455 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > > Implement the access method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943793#comment-16943793
]
Virajith Jalaparti commented on HADOOP-16005:
-
Hi [~c-w] - I wanted to follow up on this. Do you have any updates on this
JIRA? It's been a while since the last comment. We are also interested in ABFS
supporting XAttrs
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15707) Add IsActiveServlet to be used for Load Balancers
[ https://issues.apache.org/jira/browse/HADOOP-15707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598019#comment-16598019 ] Virajith Jalaparti commented on HADOOP-15707: - I don't know this part of the code much but the patch itself looks mostly good to me. Has this been tested with Azure/AWS load balancers? If so, would be great to see any info on that. > Add IsActiveServlet to be used for Load Balancers > - > > Key: HADOOP-15707 > URL: https://issues.apache.org/jira/browse/HADOOP-15707 > Project: Hadoop Common > Issue Type: New Feature > Components: common >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Major > Attachments: HADOOP-15707.000.patch, HADOOP-15707.001.patch, > HADOOP-15707.002.patch > > > Hadoop has a few services with HA setups and it is common to set them behind > Load Balancers. > We should add a way for the Load Balancers to understand what should be the > UI to show. > For example, the standby RM just redirects the requests to the active RM. > However, if both RMs are behind a Load Balancer the IP might not be reachable. > Most Load balancers have probes to check if a server reports HTTP code 200: > * > [Azure|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > * > [AWS|https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview] > Components in Hadoop (e.g., NN, RM, Router,...) should have a unified way to > report if they are active. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391856#comment-16391856 ] Virajith Jalaparti commented on HADOOP-15292: - [~ste...@apache.org], Thanks for reviewing and committing it. [~elgoiri] and [~chris.douglas], thanks for the reviews. > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 2.5.0 >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Minor > Fix For: 3.1.0 > > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch, > HADOOP-15292.002.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15292) Distcp's use of pread is slowing it down.
[
https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390047#comment-16390047
]
Virajith Jalaparti commented on HADOOP-15292:
-
[^HADOOP-15292.002.patch] fixes [~ste...@apache.org]'s and [~chris.douglas]'s
comment of seeking when {{sourceOffset != inStream.getPos()}}.
[~ste...@apache.org] {{ITestAzureNativeContractDistCp}} with this fix.
> Distcp's use of pread is slowing it down.
> -
>
> Key: HADOOP-15292
> URL: https://issues.apache.org/jira/browse/HADOOP-15292
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.5.0
>Reporter: Virajith Jalaparti
>Priority: Minor
> Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch,
> HADOOP-15292.002.patch
>
>
> Distcp currently uses positioned-reads (in
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This
> results in unnecessary overheads (new BlockReader being created on the
> client-side, multiple readBlock() calls to the Datanodes, each of which
> requires the creation of a BlockSender and an inputstream to the ReplicaInfo).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Status: Patch Available (was: Open) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 2.5.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch, > HADOOP-15292.002.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15292) Distcp's use of pread is slowing it down.
[
https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390047#comment-16390047
]
Virajith Jalaparti edited comment on HADOOP-15292 at 3/7/18 7:07 PM:
-
[^HADOOP-15292.002.patch] fixes [~ste...@apache.org]'s and [~chris.douglas]'s
comment of seeking when {{sourceOffset != inStream.getPos()}}.
[~ste...@apache.org] {{ITestAzureNativeContractDistCp}} passes after this fix
as well.
was (Author: virajith):
[^HADOOP-15292.002.patch] fixes [~ste...@apache.org]'s and [~chris.douglas]'s
comment of seeking when {{sourceOffset != inStream.getPos()}}.
[~ste...@apache.org] {{ITestAzureNativeContractDistCp}} with this fix.
> Distcp's use of pread is slowing it down.
> -
>
> Key: HADOOP-15292
> URL: https://issues.apache.org/jira/browse/HADOOP-15292
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.5.0
>Reporter: Virajith Jalaparti
>Priority: Minor
> Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch,
> HADOOP-15292.002.patch
>
>
> Distcp currently uses positioned-reads (in
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This
> results in unnecessary overheads (new BlockReader being created on the
> client-side, multiple readBlock() calls to the Datanodes, each of which
> requires the creation of a BlockSender and an inputstream to the ReplicaInfo).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Status: Open (was: Patch Available) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 2.5.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch, > HADOOP-15292.002.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: HADOOP-15292.002.patch > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 2.5.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch, > HADOOP-15292.002.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Status: Patch Available (was: Open) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: (was: HADOOP-15292.001.patch) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: HADOOP-15292.001.patch > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15292) Distcp's use of pread is slowing it down.
[
https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388928#comment-16388928
]
Virajith Jalaparti commented on HADOOP-15292:
-
bq. Probably not worth adding metrics but maybe extend the stream in the unit
test and track how many times we open it.
[~elgoiri], I extended {{TestCopyMapper#testCopyWithAppend}} to test the number
of calls to {{readBlock}} on the Datanode (this is captured by the metric
{{readsFromLocalClient}}), in [^HADOOP-15292.001.patch]. The buffer size is
set such that if pread was used, this metric would increase much more than the
number of files that are being appended to.
[~ste...@apache.org] I tested this over an internal filesystem. This issue came
up when I was using distcp to copy data from the filesystem using the tiered
HDFS feature (HDFS-9806). This particular filesystem throttled calls to open()
beyond a certain QPS. For large enough data, distcp never succeeded as it
caused way too many calls to open() (pread causes a separate InputStream to be
opened for the {{ProvidedReplica}} for each 8k chunks of data, which result in
an open() call to the remote filesystem). With this fix, Distcp runs fine, and
I don't see the throttling any more.
I think this goes towards to the "extra rigorousness" you are looking for. I am
not really setup to test this with s3.
> Distcp's use of pread is slowing it down.
> -
>
> Key: HADOOP-15292
> URL: https://issues.apache.org/jira/browse/HADOOP-15292
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.0.0
>Reporter: Virajith Jalaparti
>Priority: Minor
> Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch
>
>
> Distcp currently uses positioned-reads (in
> RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This
> results in unnecessary overheads (new BlockReader being created on the
> client-side, multiple readBlock() calls to the Datanodes, each of which
> requires the creation of a BlockSender and an inputstream to the ReplicaInfo).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Status: Open (was: Patch Available) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: HADOOP-15292.001.patch > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch, HADOOP-15292.001.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Description: Distcp currently uses positioned-reads (in RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This results in unnecessary overheads (new BlockReader being created on the client-side, multiple readBlock() calls to the Datanodes, each of which requires the creation of a BlockSender and an inputstream to the ReplicaInfo). (was: Distcp currently uses positioned-reads (in RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This results in unnecessary overheads (new BlockReader being created on the client-side, multiple readBlock() calls to the Datanodes, each of requires the creation of a BlockSender and an inputstream to the ReplicaInfo).) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Virajith Jalaparti >Priority: Minor > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of which > requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Status: Patch Available (was: Open) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of requires > the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387182#comment-16387182 ] Virajith Jalaparti commented on HADOOP-15292: - Attached patch fixes this issue by replacing the positioned read with an initial seek, and reading the remaining data directly from the open stream. > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of requires > the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: HADOOP-15292.000.patch > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of requires > the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: HADOOP-15292.000.patch > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of requires > the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15292) Distcp's use of pread is slowing it down.
[ https://issues.apache.org/jira/browse/HADOOP-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HADOOP-15292: Attachment: (was: HADOOP-15292.000.patch) > Distcp's use of pread is slowing it down. > - > > Key: HADOOP-15292 > URL: https://issues.apache.org/jira/browse/HADOOP-15292 > Project: Hadoop Common > Issue Type: Bug >Reporter: Virajith Jalaparti >Priority: Major > Attachments: HADOOP-15292.000.patch > > > Distcp currently uses positioned-reads (in > RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This > results in unnecessary overheads (new BlockReader being created on the > client-side, multiple readBlock() calls to the Datanodes, each of requires > the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15292) Distcp's use of pread is slowing it down.
Virajith Jalaparti created HADOOP-15292: --- Summary: Distcp's use of pread is slowing it down. Key: HADOOP-15292 URL: https://issues.apache.org/jira/browse/HADOOP-15292 Project: Hadoop Common Issue Type: Bug Reporter: Virajith Jalaparti Distcp currently uses positioned-reads (in RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This results in unnecessary overheads (new BlockReader being created on the client-side, multiple readBlock() calls to the Datanodes, each of requires the creation of a BlockSender and an inputstream to the ReplicaInfo). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15106) FileSystem::open(PathHandle) should throw a specific exception on validation failure
[
https://issues.apache.org/jira/browse/HADOOP-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291379#comment-16291379
]
Virajith Jalaparti commented on HADOOP-15106:
-
Minor comment -- How about moving {{InvalidPathHandleException}} above
{{IOException}} in the javadoc to be consistent with others like
{{getFileStatus}} (e.g., {{FileNotFoundException}} goes above
{{IOException}}).? Other than that, lgtm.
> FileSystem::open(PathHandle) should throw a specific exception on validation
> failure
>
>
> Key: HADOOP-15106
> URL: https://issues.apache.org/jira/browse/HADOOP-15106
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chris Douglas
>Priority: Minor
> Attachments: HADOOP-15106.00.patch, HADOOP-15106.01.patch,
> HADOOP-15106.02.patch, HADOOP-15106.03.patch
>
>
> Callers of {{FileSystem::open(PathHandle)}} cannot distinguish between I/O
> errors and an invalid handle. The signature should include a specific,
> checked exception for this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
