[jira] [Commented] (HDFS-16145) CopyListing fails with FNF exception with snapshot diff
[
https://issues.apache.org/jira/browse/HDFS-16145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751786#comment-17751786
]
Siyao Meng commented on HDFS-16145:
---
[~sodonnell] Looks like branch-3.2 would need HDFS-14869 first. After that it
is a clean backport of HDFS-16145:
{code:title=On branch-3.2, commit hash 45b7a025fc67759effd6093bbf29028c3572dd1c}
$ gcp -x fc97034b29243a0509633849de55aa734859
Auto-merging
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
[branch-3.2 fa43ae4e7d3] HDFS-14869 Copy renamed files which are not excluded
anymore by filter (#1530)
Author: aasha
Date: Fri Dec 6 17:41:25 2019 +0530
3 files changed, 185 insertions(+), 8 deletions(-)
$ gcp -x dac10fcc202ed6d1fe4bd852f57a6bbcbadd90fe
Auto-merging
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java
Auto-merging
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java
[branch-3.2 b825222d929] HDFS-16145. CopyListing fails with FNF exception with
snapshot diff. (#3234)
Author: bshashikant
Date: Wed Jul 28 10:29:00 2021 +0530
2 files changed, 231 insertions(+), 3 deletions(-)
{code}
> CopyListing fails with FNF exception with snapshot diff
> ---
>
> Key: HDFS-16145
> URL: https://issues.apache.org/jira/browse/HDFS-16145
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Distcp with snapshotdiff and with filters, marks a Rename as a delete
> opeartion on the target if the rename target is to a directory which is
> exluded by the filter. But, in cases, where files/subdirs created/modified
> prior to the Rename post the old snapshot will still be present as
> modified/created entries in the final copy list. Since, the parent diretory
> is marked for deletion, these subsequent create/modify entries should be
> ignored while building the final copy list.
> With such cases, when the final copy list is built, distcp tries to do a
> lookup for each create/modified file in the newer snapshot which will fail
> as, the parent dir is already moved to a new location in later snapshot.
>
> {code:java}
> sudo -u kms hadoop key create testkey
> hadoop fs -mkdir -p /data/gcgdlknnasg/
> hdfs crypto -createZone -keyName testkey -path /data/gcgdlknnasg/
> hadoop fs -mkdir -p /dest/gcgdlknnasg
> hdfs crypto -createZone -keyName testkey -path /dest/gcgdlknnasg
> hdfs dfs -mkdir /data/gcgdlknnasg/dir1
> hdfs dfsadmin -allowSnapshot /data/gcgdlknnasg/
> hdfs dfsadmin -allowSnapshot /dest/gcgdlknnasg/
> [root@nightly62x-1 logs]# hdfs dfs -ls -R /data/gcgdlknnasg/
> drwxrwxrwt - hdfs supergroup 0 2021-07-16 14:05
> /data/gcgdlknnasg/.Trash
> drwxr-xr-x - hdfs supergroup 0 2021-07-16 13:07
> /data/gcgdlknnasg/dir1
> [root@nightly62x-1 logs]# hdfs dfs -ls -R /dest/gcgdlknnasg/
> [root@nightly62x-1 logs]#
> hdfs dfs -put /etc/hosts /data/gcgdlknnasg/dir1/
> hdfs dfs -rm -r /data/gcgdlknnasg/dir1/
> hdfs dfs -mkdir /data/gcgdlknnasg/dir1/
> ===> Run BDR with “Abort on Snapshot Diff Failures” CHECKED now in the
> replication schedule. You get into below error and failure of the BDR job.
> 21/07/16 15:02:30 INFO distcp.DistCp: Failed to use snapshot diff -
> java.io.FileNotFoundException: File does not exist:
> /data/gcgdlknnasg/.snapshot/distcp-5-46485360-new/dir1/hosts
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1494)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1487)
> ……..
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16645) Multi inProgress segments caused "Invalid log manifest"
[
https://issues.apache.org/jira/browse/HDFS-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571585#comment-17571585
]
Siyao Meng commented on HDFS-16645:
---
Thanks [~xuzq_zander] for reporting this issue and posting the patch.
Would you give some background on how/when this issue is observed?
Or even better: is there a way to steadily repro it?
> Multi inProgress segments caused "Invalid log manifest"
> ---
>
> Key: HDFS-16645
> URL: https://issues.apache.org/jira/browse/HDFS-16645
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {code:java}
> java.lang.IllegalStateException: Invalid log manifest (log [1-?
> (in-progress)] overlaps [6-? (in-progress)])[[6-? (in-progress)], [1-?
> (in-progress)]] CommittedTxId: 0
> at
> org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest.checkState(RemoteEditLogManifest.java:62)
> at
> org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest.(RemoteEditLogManifest.java:46)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.getEditLogManifest(Journal.java:740)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16055) Quota is not preserved in snapshot INode
[
https://issues.apache.org/jira/browse/HDFS-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-16055:
--
Fix Version/s: (was: 3.3.2)
3.4.0
> Quota is not preserved in snapshot INode
>
>
> Key: HDFS-16055
> URL: https://issues.apache.org/jira/browse/HDFS-16055
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Quota feature is not preserved during snapshot creation, this causes
> {{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
> {{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if
> the quota is set before the snapshot creation:
> {code:bash}
> $ hdfs snapshotDiff /diffTest s0 .
> Difference between snapshot s0 and current directory under directory
> /diffTest:
> M .
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16055) Quota is not preserved in snapshot INode
[
https://issues.apache.org/jira/browse/HDFS-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-16055:
--
Fix Version/s: 3.3.2
Target Version/s: (was: 3.3.2)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Quota is not preserved in snapshot INode
>
>
> Key: HDFS-16055
> URL: https://issues.apache.org/jira/browse/HDFS-16055
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Quota feature is not preserved during snapshot creation, this causes
> {{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
> {{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if
> the quota is set before the snapshot creation:
> {code:bash}
> $ hdfs snapshotDiff /diffTest s0 .
> Difference between snapshot s0 and current directory under directory
> /diffTest:
> M .
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16055) Quota is not preserved in snapshot INode
[
https://issues.apache.org/jira/browse/HDFS-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-16055:
--
Status: Patch Available (was: Open)
> Quota is not preserved in snapshot INode
>
>
> Key: HDFS-16055
> URL: https://issues.apache.org/jira/browse/HDFS-16055
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Quota feature is not preserved during snapshot creation, this causes
> {{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
> {{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if
> the quota is set before the snapshot creation:
> {code:bash}
> $ hdfs snapshotDiff /diffTest s0 .
> Difference between snapshot s0 and current directory under directory
> /diffTest:
> M .
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16055) Quota is not preserved in snapshot INode
Siyao Meng created HDFS-16055:
-
Summary: Quota is not preserved in snapshot INode
Key: HDFS-16055
URL: https://issues.apache.org/jira/browse/HDFS-16055
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.3.0
Reporter: Siyao Meng
Assignee: Siyao Meng
Quota feature is not preserved during snapshot creation, this causes
{{INodeDirectory#metadataEquals}} to ALWAYS return true. Therefore,
{{snapshotDiff}} will ALWAYS return the snapshot root as modified, even if the
quota is set before the snapshot creation:
{code:bash}
$ hdfs snapshotDiff /diffTest s0 .
Difference between snapshot s0 and current directory under directory /diffTest:
M .
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15997) Implement dfsadmin -provisionSnapshotTrash -all
[ https://issues.apache.org/jira/browse/HDFS-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15997: -- Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Implement dfsadmin -provisionSnapshotTrash -all > --- > > Key: HDFS-15997 > URL: https://issues.apache.org/jira/browse/HDFS-15997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: dfsadmin >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently dfsadmin -provisionSnapshotTrash only supports creating trash root > one by one. > This jira adds -all argument to create trash root on ALL snapshottable dirs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15982) Deleted data using HTTP API should be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15982: -- Component/s: hdfs-client > Deleted data using HTTP API should be saved to the trash > > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client, httpfs, webhdfs >Reporter: Bhavik Patel >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Attachments: Screenshot 2021-04-23 at 4.19.42 PM.png, Screenshot > 2021-04-23 at 4.36.57 PM.png > > Time Spent: 7h 50m > Remaining Estimate: 0h > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > This can be helpful when the user accidentally deletes data from the Web UI. > Similarly we should provide "Skip Trash" option in HTTP API as well which > should be accessible through Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15982) Deleted data using HTTP API should be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15982: -- Labels: pull-request-available (was: incompatibleChange pull-request-available) > Deleted data using HTTP API should be saved to the trash > > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, httpfs, webhdfs >Reporter: Bhavik Patel >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Attachments: Screenshot 2021-04-23 at 4.19.42 PM.png, Screenshot > 2021-04-23 at 4.36.57 PM.png > > Time Spent: 7h 50m > Remaining Estimate: 0h > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > This can be helpful when the user accidentally deletes data from the Web UI. > Similarly we should provide "Skip Trash" option in HTTP API as well which > should be accessible through Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15982) Deleted data using HTTP API should be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15982: -- Labels: incompatibleChange pull-request-available (was: pull-request-available) > Deleted data using HTTP API should be saved to the trash > > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, httpfs, webhdfs >Reporter: Bhavik Patel >Assignee: Viraj Jasani >Priority: Major > Labels: incompatibleChange, pull-request-available > Attachments: Screenshot 2021-04-23 at 4.19.42 PM.png, Screenshot > 2021-04-23 at 4.36.57 PM.png > > Time Spent: 6h > Remaining Estimate: 0h > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > This can be helpful when the user accidentally deletes data from the Web UI. > Similarly we should provide "Skip Trash" option in HTTP API as well which > should be accessible through Web UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15997) Implement dfsadmin -provisionSnapshotTrash -all
[ https://issues.apache.org/jira/browse/HDFS-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15997: -- Description: Currently dfsadmin -provisionSnapshotTrash only supports creating trash root one by one. This jira adds -all argument to create trash root on ALL snapshottable dirs. was:Currently > Implement dfsadmin -provisionSnapshotTrash -all > --- > > Key: HDFS-15997 > URL: https://issues.apache.org/jira/browse/HDFS-15997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: dfsadmin >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently dfsadmin -provisionSnapshotTrash only supports creating trash root > one by one. > This jira adds -all argument to create trash root on ALL snapshottable dirs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15997) Implement dfsadmin -provisionSnapshotTrash -all
[ https://issues.apache.org/jira/browse/HDFS-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15997: -- Description: Currently > Implement dfsadmin -provisionSnapshotTrash -all > --- > > Key: HDFS-15997 > URL: https://issues.apache.org/jira/browse/HDFS-15997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: dfsadmin >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15997) Implement dfsadmin -provisionSnapshotTrash -all
[ https://issues.apache.org/jira/browse/HDFS-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15997: -- Status: Patch Available (was: Open) > Implement dfsadmin -provisionSnapshotTrash -all > --- > > Key: HDFS-15997 > URL: https://issues.apache.org/jira/browse/HDFS-15997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: dfsadmin >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15997) Implement dfsadmin -provisionSnapshotTrash -all
Siyao Meng created HDFS-15997: - Summary: Implement dfsadmin -provisionSnapshotTrash -all Key: HDFS-15997 URL: https://issues.apache.org/jira/browse/HDFS-15997 Project: Hadoop HDFS Issue Type: Sub-task Components: dfsadmin Reporter: Siyao Meng Assignee: Siyao Meng -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320927#comment-17320927
]
Siyao Meng edited comment on HDFS-15614 at 4/14/21, 11:31 AM:
--
Thanks for bringing this up [~ayushtkn].
{quote}
And this fails, And yep there is an ambiguity.
{quote}
The reason is that
[{{DFS#provisionSnapshotTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2984]
followed EZ counterpart's
[{{DFS#provisionEZTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2913]
implementation. {{dfs.provisionSnapshotTrash}} is not automatically called
from {{dfs.allowSnapshot}}, following what encryption zone has done similarly.
Therefore this would be the same deal for encryption zone trash root creation.
When replacing {{dfs.allowSnapshot}} calls with {{dfs.createEncryptionZone}} in
the first test case we should also find trash root inside encryption zone
missing with {{dfs.provisionSnapshotTrash}} call *alone*.
I suggest some guidelines should be posted and in javadoc adding that
allowSnapshot should better performed with dfsadmin CLI (and
createEncryptionZone if there aren't already).
{quote}
How come a client side feature that important, that can make the cluster go
down in times of critical situation like failover, Again a test to show that:
{quote}
[name
quota|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Name_Quotas]
can become an issue indeed.
I think I got your point. Maybe a better way to create those necessary Trash
dirs is to ask an admin to run dfsadmin command *manually* after flipping
{{dfs.namenode.snapshot.trashroot.enabled}} to {{true}}.
Currently we already have {{dfsadmin -provisionSnapshotTrash}} but can only be
done one by one. {{dfsadmin -provisionSnapshotTrash -all}} can be implemented
to achieve this.
Cheers,
Siyao
was (Author: smeng):
Thanks for bringing this up [~ayushtkn].
{quote}
And this fails, And yep there is an ambiguity.
{quote}
The reason is that
[{{DFS#provisionSnapshotTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2984]
followed EZ counterpart's
[{{DFS#provisionEZTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2913]
implementation. {{dfs.provisionSnapshotTrash}} is not automatically called
from {{dfs.allowSnapshot}}, following what .
Therefore this would be the same deal for encryption zone trash root creation.
When replacing {{dfs.allowSnapshot}} calls with {{dfs.createEncryptionZone}} in
the first test case we should also find trash root inside encryption zone
missing with {{dfs.provisionSnapshotTrash}} call *alone*.
I suggest some guidelines should be posted and in javadoc adding that
allowSnapshot should better performed with dfsadmin CLI (and
createEncryptionZone if there aren't already).
{quote}
How come a client side feature that important, that can make the cluster go
down in times of critical situation like failover, Again a test to show that:
{quote}
[name
quota|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Name_Quotas]
can become an issue indeed.
I think I got your point. Maybe a better way to create those necessary Trash
dirs is to ask an admin to run dfsadmin command *manually* after flipping
{{dfs.namenode.snapshot.trashroot.enabled}} to {{true}}.
Currently we already have {{dfsadmin -provisionSnapshotTrash}} but can only be
done one by one. {{dfsadmin -provisionSnapshotTrash -all}} can be implemented
to achieve this.
Cheers,
Siyao
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320927#comment-17320927
]
Siyao Meng commented on HDFS-15614:
---
Thanks for bringing this up [~ayushtkn].
{quote}
And this fails, And yep there is an ambiguity.
{quote}
The reason is that
[{{DFS#provisionSnapshotTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2984]
followed EZ counterpart's
[{{DFS#provisionEZTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2913]
implementation. {{dfs.provisionSnapshotTrash}} is not automatically called
from {{dfs.allowSnapshot}}, following what .
Therefore this would be the same deal for encryption zone trash root creation.
When replacing {{dfs.allowSnapshot}} calls with {{dfs.createEncryptionZone}} in
the first test case we should also find trash root inside encryption zone
missing with {{dfs.provisionSnapshotTrash}} call *alone*.
I suggest some guidelines should be posted and in javadoc adding that
allowSnapshot should better performed with dfsadmin CLI (and
createEncryptionZone if there aren't already).
{quote}
How come a client side feature that important, that can make the cluster go
down in times of critical situation like failover, Again a test to show that:
{quote}
[name
quota|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Name_Quotas]
can become an issue indeed.
I think I got your point. Maybe a better way to create those necessary Trash
dirs is to ask an admin to run dfsadmin command *manually* after flipping
{{dfs.namenode.snapshot.trashroot.enabled}} to {{true}}.
Currently we already have {{dfsadmin -provisionSnapshotTrash}} but can only be
done one by one. {{dfsadmin -provisionSnapshotTrash -all}} can be implemented
to achieve this.
Cheers,
Siyao
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15820) Ensure snapshot root trash provisioning happens only post safe mode exit
[
https://issues.apache.org/jira/browse/HDFS-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15820:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Ensure snapshot root trash provisioning happens only post safe mode exit
>
>
> Key: HDFS-15820
> URL: https://issues.apache.org/jira/browse/HDFS-15820
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently, on namenode startup, snapshot trash root provisioning starts as
> along with trash emptier service but namenode might not be out of safe mode
> by then. This can fail the snapshot trash dir creation thereby crashing the
> namenode. The idea here is to trigger snapshot trash provisioning only post
> safe mode exit.
> {code:java}
> 2021-02-04 11:23:47,323 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring
> NN shutdown. Shutting down immediately.
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create
> directory /upgrade/.Trash. Name node is in safe mode.
> The reported blocks 0 needs additional 1383 blocks to reach the threshold
> 0.9990 of total blocks 1385.
> The number of live datanodes 0 needs an additional 1 live datanodes to reach
> the minimum number 1.
> Safe mode will be turned off automatically once the thresholds have been
> reached. NamenodeHostName:quasar-brabeg-5.quasar-brabeg.root.hwx.site
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1542)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAndProvisionSnapshotTrashRoots(FSNamesystem.java:8269)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1939)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:967)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:936)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1673)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1740)
> 2021-02-04 11:23:47,334 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot
> create directory /upgrade/.Trash. Name node is in safe mode.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15820) Ensure snapshot root trash provisioning happens only post safe mode exit
[
https://issues.apache.org/jira/browse/HDFS-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15820:
--
Status: Patch Available (was: Open)
> Ensure snapshot root trash provisioning happens only post safe mode exit
>
>
> Key: HDFS-15820
> URL: https://issues.apache.org/jira/browse/HDFS-15820
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, on namenode startup, snapshot trash root provisioning starts as
> along with trash emptier service but namenode might not be out of safe mode
> by then. This can fail the snapshot trash dir creation thereby crashing the
> namenode. The idea here is to trigger snapshot trash provisioning only post
> safe mode exit.
> {code:java}
> 2021-02-04 11:23:47,323 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring
> NN shutdown. Shutting down immediately.
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create
> directory /upgrade/.Trash. Name node is in safe mode.
> The reported blocks 0 needs additional 1383 blocks to reach the threshold
> 0.9990 of total blocks 1385.
> The number of live datanodes 0 needs an additional 1 live datanodes to reach
> the minimum number 1.
> Safe mode will be turned off automatically once the thresholds have been
> reached. NamenodeHostName:quasar-brabeg-5.quasar-brabeg.root.hwx.site
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1542)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAndProvisionSnapshotTrashRoots(FSNamesystem.java:8269)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1939)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:967)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:936)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1673)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1740)
> 2021-02-04 11:23:47,334 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot
> create directory /upgrade/.Trash. Name node is in safe mode.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15820) Ensure snapshot root trash provisioning happens only post safe mode exit
[
https://issues.apache.org/jira/browse/HDFS-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279050#comment-17279050
]
Siyao Meng commented on HDFS-15820:
---
[~shashikant] will do.
> Ensure snapshot root trash provisioning happens only post safe mode exit
>
>
> Key: HDFS-15820
> URL: https://issues.apache.org/jira/browse/HDFS-15820
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, on namenode startup, snapshot trash root provisioning starts as
> along with trash emptier service but namenode might not be out of safe mode
> by then. This can fail the snapshot trash dir creation thereby crashing the
> namenode. The idea here is to trigger snapshot trash provisioning only post
> safe mode exit.
> {code:java}
> 2021-02-04 11:23:47,323 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring
> NN shutdown. Shutting down immediately.
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create
> directory /upgrade/.Trash. Name node is in safe mode.
> The reported blocks 0 needs additional 1383 blocks to reach the threshold
> 0.9990 of total blocks 1385.
> The number of live datanodes 0 needs an additional 1 live datanodes to reach
> the minimum number 1.
> Safe mode will be turned off automatically once the thresholds have been
> reached. NamenodeHostName:quasar-brabeg-5.quasar-brabeg.root.hwx.site
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1542)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1529)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAndProvisionSnapshotTrashRoots(FSNamesystem.java:8269)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1939)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:967)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:936)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1673)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1740)
> 2021-02-04 11:23:47,334 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot
> create directory /upgrade/.Trash. Name node is in safe mode.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15689) allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
[
https://issues.apache.org/jira/browse/HDFS-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15689:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> allow/disallowSnapshot on EZ roots shouldn't fail due to trash
> provisioning/emptiness check
> ---
>
> Key: HDFS-15689
> URL: https://issues.apache.org/jira/browse/HDFS-15689
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> h2. Background
> 1. HDFS-15607 added a feature that when
> {{dfs.namenode.snapshot.trashroot.enabled=true}}, allowSnapshot will
> automatically create a .Trash directory immediately after allowSnapshot
> operation so files deleted will be moved into the trash root inside the
> snapshottable directory.
> 2. HDFS-15539 prevents admins from disallowing snapshot if the trash root
> inside is not empty
> h2. Problem
> 1. When {{dfs.namenode.snapshot.trashroot.enabled=true}}, currently if the
> directory (to be allowed snapshot on) is an EZ root, it throws
> {{FileAlreadyExistsException}} because the trash root already exists
> (encryption zone has already created an internal trash root).
> 2. Similarly, at the moment if we disallow snapshot on an EZ root, it may
> complain that the trash root is not empty (or delete it if empty, which is
> not desired since EZ will still need it).
> h2. Solution
> 1. Let allowSnapshot succeed by not throwing {{FileAlreadyExistsException}},
> but informs the admin that the trash already exists.
> 2. Ignore {{checkTrashRootAndRemoveIfEmpty()}} check if path is EZ root.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15689) allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
[
https://issues.apache.org/jira/browse/HDFS-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15689:
--
Status: Patch Available (was: Open)
> allow/disallowSnapshot on EZ roots shouldn't fail due to trash
> provisioning/emptiness check
> ---
>
> Key: HDFS-15689
> URL: https://issues.apache.org/jira/browse/HDFS-15689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> h2. Background
> 1. HDFS-15607 added a feature that when
> {{dfs.namenode.snapshot.trashroot.enabled=true}}, allowSnapshot will
> automatically create a .Trash directory immediately after allowSnapshot
> operation so files deleted will be moved into the trash root inside the
> snapshottable directory.
> 2. HDFS-15539 prevents admins from disallowing snapshot if the trash root
> inside is not empty
> h2. Problem
> 1. When {{dfs.namenode.snapshot.trashroot.enabled=true}}, currently if the
> directory (to be allowed snapshot on) is an EZ root, it throws
> {{FileAlreadyExistsException}} because the trash root already exists
> (encryption zone has already created an internal trash root).
> 2. Similarly, at the moment if we disallow snapshot on an EZ root, it may
> complain that the trash root is not empty (or delete it if empty, which is
> not desired since EZ will still need it).
> h2. Solution
> 1. Let allowSnapshot succeed by not throwing {{FileAlreadyExistsException}},
> but informs the admin that the trash already exists.
> 2. Ignore {{checkTrashRootAndRemoveIfEmpty()}} check if path is EZ root.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15689) allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check
[
https://issues.apache.org/jira/browse/HDFS-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15689:
--
Summary: allow/disallowSnapshot on EZ roots shouldn't fail due to trash
provisioning/emptiness check (was: allow/disallowSnapshot on EZ roots
shouldn't fail due to trash root provisioning or emptiness check)
> allow/disallowSnapshot on EZ roots shouldn't fail due to trash
> provisioning/emptiness check
> ---
>
> Key: HDFS-15689
> URL: https://issues.apache.org/jira/browse/HDFS-15689
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> h2. Background
> 1. HDFS-15607 added a feature that when
> {{dfs.namenode.snapshot.trashroot.enabled=true}}, allowSnapshot will
> automatically create a .Trash directory immediately after allowSnapshot
> operation so files deleted will be moved into the trash root inside the
> snapshottable directory.
> 2. HDFS-15539 prevents admins from disallowing snapshot if the trash root
> inside is not empty
> h2. Problem
> 1. When {{dfs.namenode.snapshot.trashroot.enabled=true}}, currently if the
> directory (to be allowed snapshot on) is an EZ root, it throws
> {{FileAlreadyExistsException}} because the trash root already exists
> (encryption zone has already created an internal trash root).
> 2. Similarly, at the moment if we disallow snapshot on an EZ root, it may
> complain that the trash root is not empty (or delete it if empty, which is
> not desired since EZ will still need it).
> h2. Solution
> 1. Let allowSnapshot succeed by not throwing {{FileAlreadyExistsException}},
> but informs the admin that the trash already exists.
> 2. Ignore {{checkTrashRootAndRemoveIfEmpty()}} check if path is EZ root.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15689) allow/disallowSnapshot on EZ roots shouldn't fail due to trash root provisioning or emptiness check
Siyao Meng created HDFS-15689:
-
Summary: allow/disallowSnapshot on EZ roots shouldn't fail due to
trash root provisioning or emptiness check
Key: HDFS-15689
URL: https://issues.apache.org/jira/browse/HDFS-15689
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.4.0
Reporter: Siyao Meng
Assignee: Siyao Meng
h2. Background
1. HDFS-15607 added a feature that when
{{dfs.namenode.snapshot.trashroot.enabled=true}}, allowSnapshot will
automatically create a .Trash directory immediately after allowSnapshot
operation so files deleted will be moved into the trash root inside the
snapshottable directory.
2. HDFS-15539 prevents admins from disallowing snapshot if the trash root
inside is not empty
h2. Problem
1. When {{dfs.namenode.snapshot.trashroot.enabled=true}}, currently if the
directory (to be allowed snapshot on) is an EZ root, it throws
{{FileAlreadyExistsException}} because the trash root already exists
(encryption zone has already created an internal trash root).
2. Similarly, at the moment if we disallow snapshot on an EZ root, it may
complain that the trash root is not empty (or delete it if empty, which is not
desired since EZ will still need it).
h2. Solution
1. Let allowSnapshot succeed by not throwing {{FileAlreadyExistsException}},
but informs the admin that the trash already exists.
2. Ignore {{checkTrashRootAndRemoveIfEmpty()}} check if path is EZ root.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229496#comment-17229496
]
Siyao Meng commented on HDFS-15607:
---
Note this jira has an addendum patch that fixes a small bug where
disallowsnapshot() throws a different exception than before when called on a
*file*. To learn more, see the description for
[PR#2448|https://github.com/apache/hadoop/pull/2448].
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 (and without sticky bit) by the first user that moves a file
> to the trash. This is an issue when other users try to move files to that
> trash because they may not have the permission to move to that trash if the
> trash root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15614:
--
Fix Version/s: 3.3.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15614:
--
Fix Version/s: (was: 3.3.1)
3.4.0
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15614:
--
Status: Patch Available (was: In Progress)
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15614 started by Siyao Meng.
-
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15611: -- Status: Patch Available (was: Open) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15612) WebHDFS: Support provisionSnapshotTrash
[
https://issues.apache.org/jira/browse/HDFS-15612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15612:
--
Parent: HDFS-15477
Issue Type: Sub-task (was: Improvement)
> WebHDFS: Support provisionSnapshotTrash
> ---
>
> Key: HDFS-15612
> URL: https://issues.apache.org/jira/browse/HDFS-15612
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Priority: Minor
>
> In HDFS-15607 we are adding snapshot trash root provision command to help
> create the trash root with the right permission. But in order to add this
> feature to WebHDFS (and HttpFS), we might have to copy or move the
> client-side logic to the server since the logic for directly accessing
> WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15614:
--
Parent: HDFS-15477
Issue Type: Sub-task (was: Improvement)
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDFS-15614:
-
Assignee: Siyao Meng
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
Siyao Meng created HDFS-15614:
-
Summary: Initialize snapshot trash root during NameNode startup if
enabled
Key: HDFS-15614
URL: https://issues.apache.org/jira/browse/HDFS-15614
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Siyao Meng
This is a follow-up to HDFS-15607.
Goal:
Initialize (create) snapshot trash root for all existing snapshottable
directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to {{true}}.
So admins won't have to run {{dfsadmin -provisionTrash}} manually on all those
existing snapshottable directories.
The change is expected to land in {{FSNamesystem}}.
Discussion:
1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
client side. But in order for NN to create it at startup, the logic must (also)
be implemented on the server side as well. -- which is also a requirement by
WebHDFS (HDFS-15612).
2. Alternatively, we can provide an extra parameter to the {{-provisionTrash}}
command like: {{dfsadmin -provisionTrash -all}} to initialize/provision trash
root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 (and without sticky bit) by the first user that moves a file
> to the trash. This is an issue when other users try to move files to that
> trash because they may not have the permission to move to that trash if the
> trash root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15612) WebHDFS: Support provisionSnapshotTrash
[
https://issues.apache.org/jira/browse/HDFS-15612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15612:
--
Description:
In HDFS-15607 we are adding snapshot trash root provision command to help
create the trash root with the right permission. But in order to add this
feature to WebHDFS (and HttpFS), we might have to copy or move the client-side
logic to the server since the logic for directly accessing WebHDFS by REST API
is on the NameNode {{NamenodeWebHdfsMethods}}.
was:
In HDFS-15607 we are adding snapshot trash root provision command to help
create the trash root with the right permission. But in order to add this
feature to WebHDFS, we might have to copy or move the client-side logic to the
server since the logic for directly accessing WebHDFS by REST API is on the
NameNode {{NamenodeWebHdfsMethods}}.
> WebHDFS: Support provisionSnapshotTrash
> ---
>
> Key: HDFS-15612
> URL: https://issues.apache.org/jira/browse/HDFS-15612
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Priority: Minor
>
> In HDFS-15607 we are adding snapshot trash root provision command to help
> create the trash root with the right permission. But in order to add this
> feature to WebHDFS (and HttpFS), we might have to copy or move the
> client-side logic to the server since the logic for directly accessing
> WebHDFS by REST API is on the NameNode {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15612) WebHDFS:
Siyao Meng created HDFS-15612:
-
Summary: WebHDFS:
Key: HDFS-15612
URL: https://issues.apache.org/jira/browse/HDFS-15612
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Siyao Meng
In HDFS-15607 we are adding snapshot trash root provision command to help
create the trash root with the right permission. But in order to add this
feature to WebHDFS, we might have to copy or move the client-side logic to the
server since the logic for directly accessing WebHDFS by REST API is on the
NameNode {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15612) WebHDFS: Support provisionSnapshotTrash
[
https://issues.apache.org/jira/browse/HDFS-15612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15612:
--
Summary: WebHDFS: Support provisionSnapshotTrash (was: WebHDFS: )
> WebHDFS: Support provisionSnapshotTrash
> ---
>
> Key: HDFS-15612
> URL: https://issues.apache.org/jira/browse/HDFS-15612
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Priority: Minor
>
> In HDFS-15607 we are adding snapshot trash root provision command to help
> create the trash root with the right permission. But in order to add this
> feature to WebHDFS, we might have to copy or move the client-side logic to
> the server since the logic for directly accessing WebHDFS by REST API is on
> the NameNode {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Status: Patch Available (was: In Progress)
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 (and without sticky bit) by the first user that moves a file
> to the trash. This is an issue when other users try to move files to that
> trash because they may not have the permission to move to that trash if the
> trash root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15609) Implement hdfs getconf -serverDefaults
[
https://issues.apache.org/jira/browse/HDFS-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15609:
--
Description:
There doesn't seem to be an existing way to easily verify whether a Hadoop/HDFS
client correctly receives the FsServerDefaults from the NameNode.
Here I propose extending {{hadoop getconf}} command with parameter
{{-serverDefaults}}. When executed, the client shall get {{FsServerDefaults}}
from the specified NameNode and print the configuration values out.
was:
There doesn't seem to be a way to verify whether a Hadoop/HDFS client correctly
receives the FsServerDefaults from the NameNode.
Here I propose extending {{hadoop getconf}} command with parameter
{{-serverDefaults}}. When executed, the client shall get {{FsServerDefaults}}
from the specified NameNode and print the configuration values out.
> Implement hdfs getconf -serverDefaults
> --
>
> Key: HDFS-15609
> URL: https://issues.apache.org/jira/browse/HDFS-15609
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> There doesn't seem to be an existing way to easily verify whether a
> Hadoop/HDFS client correctly receives the FsServerDefaults from the NameNode.
> Here I propose extending {{hadoop getconf}} command with parameter
> {{-serverDefaults}}. When executed, the client shall get {{FsServerDefaults}}
> from the specified NameNode and print the configuration values out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15609) Implement hdfs getconf -serverDefaults
Siyao Meng created HDFS-15609:
-
Summary: Implement hdfs getconf -serverDefaults
Key: HDFS-15609
URL: https://issues.apache.org/jira/browse/HDFS-15609
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.3.0, 3.4.0
Reporter: Siyao Meng
Assignee: Siyao Meng
There doesn't seem to be a way to verify whether a Hadoop/HDFS client correctly
receives the FsServerDefaults from the NameNode.
Here I propose extending {{hadoop getconf}} command with parameter
{{-serverDefaults}}. When executed, the client shall get {{FsServerDefaults}}
from the specified NameNode and print the configuration values out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 (and without sticky bit) by the first user that moves a file to
the trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories
({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and when a user
performing move to trash operations doesn't have admin permissions.
Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
enabled (similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled
({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
{{.Trash}} directory anyway? Or should we ask the admin to provision trash
manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
existing cluster?
- If the cluster is just upgraded, we need to provision trash manually anyway.
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories
({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and when a user
performing move to trash operations doesn't have admin permissions.
Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
enabled (similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled
({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
{{.Trash}} directory anyway? Or should we ask the admin to provision trash
manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
existing cluster?
- If the cluster is just upgraded, we need to provision trash manually anyway.
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 (and without sticky bit) by the first user that moves a file
> to the trash. This is an issue when other users try to move files to that
> trash because they may not have the permission to move to that trash if the
> trash root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories
({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and when a user
performing move to trash operations doesn't have admin permissions.
Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
enabled (similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled
({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
{{.Trash}} directory anyway? Or should we ask the admin to provision trash
manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
existing cluster?
- If the cluster is just upgraded, we need to provision trash manually anyway.
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories
({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and when a user
performing move to trash operations doesn't have admin permissions.
Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
enabled (similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled
({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
{{.Trash}} directory anyway? Or should we ask the admin to provision trash
manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
existing cluster?
- If the cluster is just upgraded, we need to provision trash manually anyway.
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that moves a file to the
> trash. This is an issue when other users try to move files to that trash
> because they may not have the permission to move to that trash if the trash
> root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories
({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and when a user
performing move to trash operations doesn't have admin permissions.
Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
enabled (similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled
({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
{{.Trash}} directory anyway? Or should we ask the admin to provision trash
manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
existing cluster?
- If the cluster is just upgraded, we need to provision trash manually anyway.
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that moves a file to the
> trash. This is an issue when other users try to move files to that trash
> because they may not have the permission to move to that trash if the trash
> root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories ({{dfs.namenode.snapshot.trashroot.enabled}} set to true), and
> when a user performing move to trash operations doesn't have admin
> permissions.
> Solution: Create a {{.Trash}} directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled
> ({{dfs.namenode.snapshot.trashroot.enabled}} set to false), create the
> {{.Trash}} directory anyway? Or should we ask the admin to provision trash
> manually after enabling {{dfs.namenode.snapshot.trashroot.enabled}} on an
> existing cluster?
> - If the cluster is just upgraded, we need to provision trash manually anyway.
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that moves a file to the
trash. This is an issue when other users try to move files to that trash
because they may not have the permission to move to that trash if the trash
root is shared. -- in this case, snapshottable directories.
This only affects users when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to move to
trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that moves a file to the
> trash. This is an issue when other users try to move files to that trash
> because they may not have the permission to move to that trash if the trash
> root is shared. -- in this case, snapshottable directories.
> This only affects users when trash is enabled inside snapshottable
> directories, and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Create trash dir when allowing snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Summary: Create trash dir when allowing snapshottable dir (was: Trash
directory should be created in advance for snapshottable directory)
> Create trash dir when allowing snapshottable dir
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Parent: HDFS-15477
Issue Type: Sub-task (was: Bug)
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Affects Version/s: 3.4.0
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Target Version/s: 3.4.0
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to move to
trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. just remove the
.Trash directory when disallowing snapshot on a dir if it is empty?
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to move to
trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. just remove the
> .Trash directory when disallowing snapshot on a dir if it is empty?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to move to
trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to the trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to move
> to trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. i.e. remove the
> empty .Trash directory when disallowing snapshot on a dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15607 started by Siyao Meng.
-
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to the
> trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. i.e. remove the
> empty .Trash directory when disallowing snapshot on a dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Description:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits by the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash because the second user might not have the permission to the trash.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(similar solution as HDFS-10324).
Also need to deal with some corner cases:
1. even when the snapshottable directory trash root config is not enabled,
create the {{.Trash}} directory anyway?
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
was:
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits of the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash cause the user might not have the permission to do so.
Rationale is similar to HDFS-10324 except that one was for the encryption zone.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(same as HDFS-10324).
Also, need to deal with some corner cases:
1. When the snapshottable directory trash root config is not enabled
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits by the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash because the second user might not have the permission to the
> trash.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (similar solution as HDFS-10324).
> Also need to deal with some corner cases:
> 1. even when the snapshottable directory trash root config is not enabled,
> create the {{.Trash}} directory anyway?
> 2. When immediately disallowing trash, it shouldn't fail. i.e. remove the
> empty .Trash directory when disallowing snapshot on a dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
Siyao Meng created HDFS-15607:
-
Summary: Trash directory should be created in advance for
snapshottable directory
Key: HDFS-15607
URL: https://issues.apache.org/jira/browse/HDFS-15607
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Siyao Meng
Assignee: Siyao Meng
In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
permission 700 without sticky bits of the first user that attempts to move a
file to the trash. This causes issue when a second user tries to move a file to
that trash cause the user might not have the permission to do so.
Rationale is similar to HDFS-10324 except that one was for the encryption zone.
Only affects the user when trash is enabled inside snapshottable directories,
and when the user doesn't have admin permissions.
Solution: Create a .Trash directory with 777 permission and sticky bits enabled
(same as HDFS-10324).
Also, need to deal with some corner cases:
1. When the snapshottable directory trash root config is not enabled
2. When immediately disallowing trash, it shouldn't fail. i.e. remove the empty
.Trash directory when disallowing snapshot on a dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15607) Trash directory should be created in advance for snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15607:
--
Component/s: hdfs
> Trash directory should be created in advance for snapshottable directory
>
>
> Key: HDFS-15607
> URL: https://issues.apache.org/jira/browse/HDFS-15607
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> In {{TrashPolicyDefault}}, the {{.Trash}} directory will be created with
> permission 700 without sticky bits of the first user that attempts to move a
> file to the trash. This causes issue when a second user tries to move a file
> to that trash cause the user might not have the permission to do so.
> Rationale is similar to HDFS-10324 except that one was for the encryption
> zone.
> Only affects the user when trash is enabled inside snapshottable directories,
> and when the user doesn't have admin permissions.
> Solution: Create a .Trash directory with 777 permission and sticky bits
> enabled (same as HDFS-10324).
> Also, need to deal with some corner cases:
> 1. When the snapshottable directory trash root config is not enabled
> 2. When immediately disallowing trash, it shouldn't fail. i.e. remove the
> empty .Trash directory when disallowing snapshot on a dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15513) Allow client to query snapshot status on one directory
[
https://issues.apache.org/jira/browse/HDFS-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200250#comment-17200250
]
Siyao Meng commented on HDFS-15513:
---
Hi [~jianghuazhu], thanks for the comment.
The goal here is to reduce the overhead of getting the entire snapshot listing,
so we can query the snapshottable status of only the list of directories the
client is interested in.
{{DFSClient#getSnapshotListing}} eventually calls into
{{FSNamesystem#getSnapshottableDirListing}}, the latter of which I mentioned in
the description. It always returns the full listing. The overhead grows when
there are more snapshottable directories.
> Allow client to query snapshot status on one directory
> --
>
> Key: HDFS-15513
> URL: https://issues.apache.org/jira/browse/HDFS-15513
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Priority: Major
>
> Alternatively, we can allow the client to query snapshot status on *a list
> of* given directories by the client. Thoughts?
> Rationale:
> At the moment, we could only retrieve the full list of snapshottable
> directories with
> [{{getSnapshottableDirListing()}}|https://github.com/apache/hadoop/blob/233619a0a462ae2eb7e7253b6bb8ae48eaa5eb19/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L6986-L6994].
> This leads to the inefficiency In HDFS-15492 that we have to get the
> *entire* list of snapshottable directory to check if a file being deleted is
> inside a snapshottable directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15539:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15563:
--
Description:
Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
enabled.
Root cause analysis:
{{SnapshottableDirectoryStatus}} paths retrived inside
{{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some
directories with the same path prefix to be mistakenly classified as
snapshottable directory.
Thanks [~shashikant] for the test case addition.
---
Repro:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
was:
Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
enabled.
Root cause analysis:
{{DFSClient#getSnapshotRoot}} check loop
Thanks [~shashikant] for the test case addition.
---
Repro:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{SnapshottableDirectoryStatus}} paths retrived inside
> {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some
> directories with the same path prefix to be mistakenly classified as
> snapshottable directory.
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
>
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15563:
--
Status: Patch Available (was: Open)
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.4.0
>
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{DFSClient#getSnapshotRoot}} check loop
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15563:
--
Description:
Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
enabled.
Root cause analysis:
{{DFSClient#getSnapshotRoot}} check loop
Thanks [~shashikant] for the test case addition.
---
Repro:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
was:
Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
enabled.
Root cause analysis:
DFSClient#getSnapshotRoot
Thanks [~shashikant] for the test case addition.
---
Repro:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.4.0
>
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> {{DFSClient#getSnapshotRoot}} check loop
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15563:
--
Description:
Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
enabled.
Root cause analysis:
DFSClient#getSnapshotRoot
Thanks [~shashikant] for the test case addition.
---
Repro:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
was:
{code:java}
1. snapshottable directory present in the cluster
hdfs lsSnapshottableDir
drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 /user/hrt_4/newdir/subdir2.
Created a new directory outside snapshottable directory
hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it failed
hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.
{code}
"*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
"*/user/hrt_4/newdir/subdir/.Trash*"
as clear from the msg here:
{noformat}
rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
/user/hrt_4/newdir/subdir2 and dest
/user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are not
under the same snapshot root.{noformat}
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.4.0
>
>
> Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is
> enabled.
> Root cause analysis:
> DFSClient#getSnapshotRoot
> Thanks [~shashikant] for the test case addition.
> ---
> Repro:
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[
https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15563:
--
Summary: Incorrect getTrashRoot return value when a non-snapshottable dir
prefix matches the path of a snapshottable dir (was: getTrashRoot() location
can be wrong when the non-snapshottable directory contains the name of the
snapshottable directory in its name )
> Incorrect getTrashRoot return value when a non-snapshottable dir prefix
> matches the path of a snapshottable dir
> ---
>
> Key: HDFS-15563
> URL: https://issues.apache.org/jira/browse/HDFS-15563
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 3.4.0
>Reporter: Nilotpal Nandi
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.4.0
>
>
> {code:java}
> 1. snapshottable directory present in the cluster
> hdfs lsSnapshottableDir
> drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2
> drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536
> /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable
> directory
> hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it
> failed
> hdfs dfs -rm -r /user/hrt_4/newdir/subdir2
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.
> {code}
> "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be
> "*/user/hrt_4/newdir/subdir/.Trash*"
> as clear from the msg here:
> {noformat}
> rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source
> /user/hrt_4/newdir/subdir2 and dest
> /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are
> not under the same snapshot root.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15539:
--
Status: Patch Available (was: Open)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15539:
--
Parent: HDFS-15477
Issue Type: Sub-task (was: Improvement)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its snapshot root is not empty
Siyao Meng created HDFS-15539:
-
Summary: When disallowing snapshot on a dir, throw exception if
its snapshot root is not empty
Key: HDFS-15539
URL: https://issues.apache.org/jira/browse/HDFS-15539
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Reporter: Siyao Meng
Assignee: Siyao Meng
When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
trash root in that dir anymore (if any). The risk is the trash root will be
left there forever.
We need to throw an exception there and prompt the user to clean up or rename
the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15539:
--
Summary: When disallowing snapshot on a dir, throw exception if its trash
root is not empty (was: When disallowing snapshot on a dir, throw exception if
its snapshot root is not empty)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15526) Tests in TestOzoneFileSystem should use the existing MiniOzoneCluster
[
https://issues.apache.org/jira/browse/HDFS-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15526:
--
Status: Patch Available (was: Open)
> Tests in TestOzoneFileSystem should use the existing MiniOzoneCluster
> -
>
> Key: HDFS-15526
> URL: https://issues.apache.org/jira/browse/HDFS-15526
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Trivial
>
> In HDDS-2883, [~adoroszlai] made a change that greatly reduces the run time
> of the test suite {{TestOzoneFileSystem}} by sharing one {{MiniOzoneCluster}}
> among the tests.
> But 4 new tests have been added since and are not sharing that
> {{MiniOzoneCluster}}.
> I am able to cut down the run time of {{TestOzoneFileSystem}} from 3m18s to
> 1m2s on my Mac. It would only save more run time on GitHub Workflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15526) Tests in TestOzoneFileSystem should use the existing MiniOzoneCluster
Siyao Meng created HDFS-15526:
-
Summary: Tests in TestOzoneFileSystem should use the existing
MiniOzoneCluster
Key: HDFS-15526
URL: https://issues.apache.org/jira/browse/HDFS-15526
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Reporter: Siyao Meng
Assignee: Siyao Meng
In HDDS-2883, [~adoroszlai] made a change that greatly reduces the run time of
the test suite {{TestOzoneFileSystem}} by sharing one {{MiniOzoneCluster}}
among the tests.
But 4 new tests have been added since and are not sharing that
{{MiniOzoneCluster}}.
I am able to cut down the run time of {{TestOzoneFileSystem}} from 3m18s to
1m2s on my Mac. It would only save more run time on GitHub Workflow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15525) Make trash root inside each snapshottable directory for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15525:
--
Status: Patch Available (was: Open)
> Make trash root inside each snapshottable directory for WebHDFS
> ---
>
> Key: HDFS-15525
> URL: https://issues.apache.org/jira/browse/HDFS-15525
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: webhdfs
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Same objective as HDFS-15492, but for WebHDFS due to it having a different
> call path for {{getTrashRoot}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15525) Make trash root inside each snapshottable directory for WebHDFS
Siyao Meng created HDFS-15525:
-
Summary: Make trash root inside each snapshottable directory for
WebHDFS
Key: HDFS-15525
URL: https://issues.apache.org/jira/browse/HDFS-15525
Project: Hadoop HDFS
Issue Type: Sub-task
Components: webhdfs
Affects Versions: 3.3.0
Reporter: Siyao Meng
Assignee: Siyao Meng
Same objective as HDFS-15492, but for WebHDFS due to it having a different call
path for {{getTrashRoot}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Fix Version/s: 3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.4.0
>
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enabled}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15513) Allow client to query snapshot status on one directory
[
https://issues.apache.org/jira/browse/HDFS-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15513:
--
Description:
Alternatively, we can allow the client to query snapshot status on *a list of*
given directories by the client. Thoughts?
Rationale:
At the moment, we could only retrieve the full list of snapshottable
directories with
[{{getSnapshottableDirListing()}}|https://github.com/apache/hadoop/blob/233619a0a462ae2eb7e7253b6bb8ae48eaa5eb19/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L6986-L6994].
This leads to the inefficiency In HDFS-15492 that we have to get the *entire*
list of snapshottable directory to check if a file being deleted is inside a
snapshottable directory.
was:
Alternatively, we can allow the client to query snapshot status on *a list of*
directories, if necessary. Thoughts?
Rationale:
At the moment, we could only retrieve the full list of snapshottable
directories with
[{{getSnapshottableDirListing()}}|https://github.com/apache/hadoop/blob/233619a0a462ae2eb7e7253b6bb8ae48eaa5eb19/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L6986-L6994].
This leads to the inefficiency In HDFS-15492 that we have to get the *entire*
list of snapshottable directory to check if a file being deleted is inside a
snapshottable directory.
> Allow client to query snapshot status on one directory
> --
>
> Key: HDFS-15513
> URL: https://issues.apache.org/jira/browse/HDFS-15513
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.3.0
>Reporter: Siyao Meng
>Priority: Major
>
> Alternatively, we can allow the client to query snapshot status on *a list
> of* given directories by the client. Thoughts?
> Rationale:
> At the moment, we could only retrieve the full list of snapshottable
> directories with
> [{{getSnapshottableDirListing()}}|https://github.com/apache/hadoop/blob/233619a0a462ae2eb7e7253b6bb8ae48eaa5eb19/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L6986-L6994].
> This leads to the inefficiency In HDFS-15492 that we have to get the
> *entire* list of snapshottable directory to check if a file being deleted is
> inside a snapshottable directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15513) Allow client to query snapshot status on one directory
Siyao Meng created HDFS-15513:
-
Summary: Allow client to query snapshot status on one directory
Key: HDFS-15513
URL: https://issues.apache.org/jira/browse/HDFS-15513
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs, hdfs-client
Affects Versions: 3.3.0
Reporter: Siyao Meng
Alternatively, we can allow the client to query snapshot status on *a list of*
directories, if necessary. Thoughts?
Rationale:
At the moment, we could only retrieve the full list of snapshottable
directories with
[{{getSnapshottableDirListing()}}|https://github.com/apache/hadoop/blob/233619a0a462ae2eb7e7253b6bb8ae48eaa5eb19/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L6986-L6994].
This leads to the inefficiency In HDFS-15492 that we have to get the *entire*
list of snapshottable directory to check if a file being deleted is inside a
snapshottable directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Status: Patch Available (was: In Progress)
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enabled}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Description:
We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside one
snapshottable directories are moved outside of it. The most common case of this
is when trash is enabled and user deletes some file via the command line
without skipTrash.
This jira aims to make a trash root for each snapshottable directory, same as
how encryption zone behaves at the moment.
This will make trash cleanup a little bit more expensive on the NameNode as it
will be to iterate all trash roots. But should be fine as long as there aren't
many snapshottable directories.
I could make this improvement as an option and disable it by default if needed,
such as {{dfs.namenode.snapshot.trashroot.enabled}}
One small caveat though, when disabling (disallowing) snapshot on the
snapshottable directory when this improvement is in place. The client should
merge the snapshottable directory's trash with that user's trash to ensure
proper trash cleanup.
was:
We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside one
snapshottable directories are moved outside of it. The most common case of this
is when trash is enabled and user deletes some file via the command line
without skipTrash.
This jira aims to make a trash root for each snapshottable directory, same as
how encryption zone behaves at the moment.
This will make trash cleanup a little bit more expensive on the NameNode as it
will be to iterate all trash roots. But should be fine as long as there aren't
many snapshottable directories.
I could make this improvement as an option and disable it by default if needed,
such as {{dfs.namenode.snapshot.trashroot.enable}}
One small caveat though, when disabling (disallowing) snapshot on the
snapshottable directory when this improvement is in place. The client should
merge the snapshottable directory's trash with that user's trash to ensure
proper trash cleanup.
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enabled}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15495) Decommissioning a DataNode with corrupted EC files should not be blocked indefinitely
[
https://issues.apache.org/jira/browse/HDFS-15495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167526#comment-17167526
]
Siyao Meng edited comment on HDFS-15495 at 7/29/20, 10:20 PM:
--
Awesome! Thanks [~sodonnell] for the writeup and unit test.
I tried your UT and it works - I mean it failed because the decom is hung. So
we can confirm this is still the case on trunk.
was (Author: smeng):
Awesome! Thanks [~sodonnell] for the writeup and unit test.
I tried your UT and it works - I mean it failed because the decom is hang. So
we can confirm this is still the case on trunk.
> Decommissioning a DataNode with corrupted EC files should not be blocked
> indefinitely
> -
>
> Key: HDFS-15495
> URL: https://issues.apache.org/jira/browse/HDFS-15495
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement, ec
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
> HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
> HDFS-15186).
> When there's an EC file marked as corrupted on NN, if the admin tries to
> decommission a DataNode having one of the remaining blocks of the corrupted
> EC file, *the decom will never finish* unless the file is recovered by
> putting the missing blocks back in:
> {code:title=The endless DatanodeAdminManager check loop, every 30s}
> 2020-07-23 16:36:12,805 TRACE blockmanagement.DatanodeAdminManager: Processed
> 0 blocks so far this tick
> 2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager:
> Processing Decommission In Progress node 127.0.1.7:5007
> 2020-07-23 16:36:12,806 TRACE blockmanagement.DatanodeAdminManager: Block
> blk_-9223372036854775728_1013 numExpected=9, numLive=4
> 2020-07-23 16:36:12,806 INFO BlockStateChange: Block:
> blk_-9223372036854775728_1013, Expected Replicas: 9, live replicas: 4,
> corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
> maintenance replicas: 0, live entering maintenance replicas: 0, excess
> replicas: 0, Is Open File: false, Datanodes having this block:
> 127.0.1.12:5012 127.0.1.10:5010 127.0.1.8:5008 127.0.1.11:5011 127.0.1.7:5007
> , Current Datanode: 127.0.1.7:5007, Is current datanode decommissioning:
> true, Is current datanode entering maintenance: false
> 2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Node
> 127.0.1.7:5007 still has 1 blocks to replicate before it is a candidate to
> finish Decommission In Progress.
> 2020-07-23 16:36:12,806 INFO blockmanagement.DatanodeAdminManager: Checked 1
> blocks and 1 nodes this tick
> {code}
> "Corrupted" file here meaning the EC file doesn't have enough EC blocks in
> the block group to be reconstructed. e.g. for {{RS-6-3-1024k}}, when there
> are less than 6 blocks for an EC file, the file can no longer be retrieved
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15495) Decommissioning a DataNode with corrupted EC files should not be blocked indefinitely
[
https://issues.apache.org/jira/browse/HDFS-15495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15495:
--
Description:
Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
HDFS-15186).
When there's an EC file marked as corrupted on NN, if the admin tries to
decommission a DataNode having one of the remaining blocks of the corrupted EC
file, *the decom will never finish* unless the file is recovered by putting the
missing blocks back in:
{code:title=The endless DatanodeAdminManager check loop, every 30s}
2020-07-23 16:36:12,805 TRACE blockmanagement.DatanodeAdminManager: Processed 0
blocks so far this tick
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Processing
Decommission In Progress node 127.0.1.7:5007
2020-07-23 16:36:12,806 TRACE blockmanagement.DatanodeAdminManager: Block
blk_-9223372036854775728_1013 numExpected=9, numLive=4
2020-07-23 16:36:12,806 INFO BlockStateChange: Block:
blk_-9223372036854775728_1013, Expected Replicas: 9, live replicas: 4, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
maintenance replicas: 0, live entering maintenance replicas: 0, excess
replicas: 0, Is Open File: false, Datanodes having this block: 127.0.1.12:5012
127.0.1.10:5010 127.0.1.8:5008 127.0.1.11:5011 127.0.1.7:5007 , Current
Datanode: 127.0.1.7:5007, Is current datanode decommissioning: true, Is current
datanode entering maintenance: false
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Node
127.0.1.7:5007 still has 1 blocks to replicate before it is a candidate to
finish Decommission In Progress.
2020-07-23 16:36:12,806 INFO blockmanagement.DatanodeAdminManager: Checked 1
blocks and 1 nodes this tick
{code}
"Corrupted" file here meaning the EC file doesn't have enough EC blocks in the
block group to be reconstructed. e.g. for {{RS-6-3-1024k}}, when there are less
than 6 blocks for an EC file, the file can no longer be retrieved correctly.
was:
Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
HDFS-15186).
When there's an EC file marked as corrupted on NN, if the admin tries to
decommission a DataNode having one of the remaining blocks of the corrupted EC
file, *the decom will never finish* unless the file is recovered by putting the
missing blocks back in:
{code:title=The endless DatanodeAdminManager check loop, every 30s}
2020-07-23 16:36:12,805 TRACE blockmanagement.DatanodeAdminManager: Processed 0
blocks so far this tick
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Processing
Decommission In Progress node 127.0.1.7:5007
2020-07-23 16:36:12,806 TRACE blockmanagement.DatanodeAdminManager: Block
blk_-9223372036854775728_1013 numExpected=9, numLive=4
2020-07-23 16:36:12,806 INFO BlockStateChange: Block:
blk_-9223372036854775728_1013, Expected Replicas: 9, live replicas: 4, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
maintenance replicas: 0, live entering maintenance replicas: 0, excess
replicas: 0, Is Open File: false, Datanodes having this block: 127.0.1.12:5012
127.0.1.10:5010 127.0.1.8:5008 127.0.1.11:5011 127.0.1.7:5007 , Current
Datanode: 127.0.1.7:5007, Is current datanode decommissioning: true, Is current
datanode entering maintenance: false
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Node
127.0.1.7:5007 still has 1 blocks to replicate before it is a candidate to
finish Decommission In Progress.
2020-07-23 16:36:12,806 INFO blockmanagement.DatanodeAdminManager: Checked 1
blocks and 1 nodes this tick
{code}
"Corrupted" file here meaning the EC file doesn't have enough EC blocks in the
block group to be reconstructed. e.g. for {{RS-6-3-1024k}}, when there are less
than 6 blocks for an EC file, the file can no longer be retrieved correctly.
Will check on trunk as well soon.
> Decommissioning a DataNode with corrupted EC files should not be blocked
> indefinitely
> -
>
> Key: HDFS-15495
> URL: https://issues.apache.org/jira/browse/HDFS-15495
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement, ec
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
> HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
> HDFS-15186).
> When there's an EC file marked as corrupted on NN, if the admin tries to
> decommission a DataNode having one of the remaining blocks of the corrupted
> EC file, *the decom will never
[jira] [Commented] (HDFS-15495) Decommissioning a DataNode with corrupted EC files should not be blocked indefinitely
[
https://issues.apache.org/jira/browse/HDFS-15495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167526#comment-17167526
]
Siyao Meng commented on HDFS-15495:
---
Awesome! Thanks [~sodonnell] for the writeup and unit test.
I tried your UT and it works - I mean it failed because the decom is hang. So
we can confirm this is still the case on trunk.
> Decommissioning a DataNode with corrupted EC files should not be blocked
> indefinitely
> -
>
> Key: HDFS-15495
> URL: https://issues.apache.org/jira/browse/HDFS-15495
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement, ec
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
> HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
> HDFS-15186).
> When there's an EC file marked as corrupted on NN, if the admin tries to
> decommission a DataNode having one of the remaining blocks of the corrupted
> EC file, *the decom will never finish* unless the file is recovered by
> putting the missing blocks back in:
> {code:title=The endless DatanodeAdminManager check loop, every 30s}
> 2020-07-23 16:36:12,805 TRACE blockmanagement.DatanodeAdminManager: Processed
> 0 blocks so far this tick
> 2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager:
> Processing Decommission In Progress node 127.0.1.7:5007
> 2020-07-23 16:36:12,806 TRACE blockmanagement.DatanodeAdminManager: Block
> blk_-9223372036854775728_1013 numExpected=9, numLive=4
> 2020-07-23 16:36:12,806 INFO BlockStateChange: Block:
> blk_-9223372036854775728_1013, Expected Replicas: 9, live replicas: 4,
> corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
> maintenance replicas: 0, live entering maintenance replicas: 0, excess
> replicas: 0, Is Open File: false, Datanodes having this block:
> 127.0.1.12:5012 127.0.1.10:5010 127.0.1.8:5008 127.0.1.11:5011 127.0.1.7:5007
> , Current Datanode: 127.0.1.7:5007, Is current datanode decommissioning:
> true, Is current datanode entering maintenance: false
> 2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Node
> 127.0.1.7:5007 still has 1 blocks to replicate before it is a candidate to
> finish Decommission In Progress.
> 2020-07-23 16:36:12,806 INFO blockmanagement.DatanodeAdminManager: Checked 1
> blocks and 1 nodes this tick
> {code}
> "Corrupted" file here meaning the EC file doesn't have enough EC blocks in
> the block group to be reconstructed. e.g. for {{RS-6-3-1024k}}, when there
> are less than 6 blocks for an EC file, the file can no longer be retrieved
> correctly.
> Will check on trunk as well soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15495) Decommissioning a DataNode with corrupted EC files should not be blocked indefinitely
Siyao Meng created HDFS-15495:
-
Summary: Decommissioning a DataNode with corrupted EC files should
not be blocked indefinitely
Key: HDFS-15495
URL: https://issues.apache.org/jira/browse/HDFS-15495
Project: Hadoop HDFS
Issue Type: Improvement
Components: block placement, ec
Affects Versions: 3.0.0
Reporter: Siyao Meng
Assignee: Siyao Meng
Originally discovered in patched CDH 6.2.1 (with a bunch of EC fixes:
HDFS-14699, HDFS-14849, HDFS-14847, HDFS-14920, HDFS-14768, HDFS-14946,
HDFS-15186).
When there's an EC file marked as corrupted on NN, if the admin tries to
decommission a DataNode having one of the remaining blocks of the corrupted EC
file, *the decom will never finish* unless the file is recovered by putting the
missing blocks back in:
{code:title=The endless DatanodeAdminManager check loop, every 30s}
2020-07-23 16:36:12,805 TRACE blockmanagement.DatanodeAdminManager: Processed 0
blocks so far this tick
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Processing
Decommission In Progress node 127.0.1.7:5007
2020-07-23 16:36:12,806 TRACE blockmanagement.DatanodeAdminManager: Block
blk_-9223372036854775728_1013 numExpected=9, numLive=4
2020-07-23 16:36:12,806 INFO BlockStateChange: Block:
blk_-9223372036854775728_1013, Expected Replicas: 9, live replicas: 4, corrupt
replicas: 0, decommissioned replicas: 0, decommissioning replicas: 1,
maintenance replicas: 0, live entering maintenance replicas: 0, excess
replicas: 0, Is Open File: false, Datanodes having this block: 127.0.1.12:5012
127.0.1.10:5010 127.0.1.8:5008 127.0.1.11:5011 127.0.1.7:5007 , Current
Datanode: 127.0.1.7:5007, Is current datanode decommissioning: true, Is current
datanode entering maintenance: false
2020-07-23 16:36:12,806 DEBUG blockmanagement.DatanodeAdminManager: Node
127.0.1.7:5007 still has 1 blocks to replicate before it is a candidate to
finish Decommission In Progress.
2020-07-23 16:36:12,806 INFO blockmanagement.DatanodeAdminManager: Checked 1
blocks and 1 nodes this tick
{code}
"Corrupted" file here meaning the EC file doesn't have enough EC blocks in the
block group to be reconstructed. e.g. for {{RS-6-3-1024k}}, when there are less
than 6 blocks for an EC file, the file can no longer be retrieved correctly.
Will check on trunk as well soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-15492 started by Siyao Meng.
-
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enable}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Affects Version/s: 3.2.1
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enable}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Target Version/s: 3.3.1
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Affects Versions: 3.2.1
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enable}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15492) Make trash root inside each snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-15492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-15492:
--
Component/s: hdfs-client
hdfs
> Make trash root inside each snapshottable directory
> ---
>
> Key: HDFS-15492
> URL: https://issues.apache.org/jira/browse/HDFS-15492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, hdfs-client
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside
> one snapshottable directories are moved outside of it. The most common case
> of this is when trash is enabled and user deletes some file via the command
> line without skipTrash.
> This jira aims to make a trash root for each snapshottable directory, same as
> how encryption zone behaves at the moment.
> This will make trash cleanup a little bit more expensive on the NameNode as
> it will be to iterate all trash roots. But should be fine as long as there
> aren't many snapshottable directories.
> I could make this improvement as an option and disable it by default if
> needed, such as {{dfs.namenode.snapshot.trashroot.enable}}
> One small caveat though, when disabling (disallowing) snapshot on the
> snapshottable directory when this improvement is in place. The client should
> merge the snapshottable directory's trash with that user's trash to ensure
> proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15492) Make trash root inside each snapshottable directory
Siyao Meng created HDFS-15492:
-
Summary: Make trash root inside each snapshottable directory
Key: HDFS-15492
URL: https://issues.apache.org/jira/browse/HDFS-15492
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Siyao Meng
Assignee: Siyao Meng
We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside one
snapshottable directories are moved outside of it. The most common case of this
is when trash is enabled and user deletes some file via the command line
without skipTrash.
This jira aims to make a trash root for each snapshottable directory, same as
how encryption zone behaves at the moment.
This will make trash cleanup a little bit more expensive on the NameNode as it
will be to iterate all trash roots. But should be fine as long as there aren't
many snapshottable directories.
I could make this improvement as an option and disable it by default if needed,
such as {{dfs.namenode.snapshot.trashroot.enable}}
One small caveat though, when disabling (disallowing) snapshot on the
snapshottable directory when this improvement is in place. The client should
merge the snapshottable directory's trash with that user's trash to ensure
proper trash cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15462) Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15462: -- Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml > - > > Key: HDFS-15462 > URL: https://issues.apache.org/jira/browse/HDFS-15462 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Fix For: 3.4.0 > > > HDFS-15394 added the existing impls in core-default.xml except ofs. Let's add > ofs to core-default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15462) Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15462: -- Component/s: viewfsOverloadScheme viewfs configuration Target Version/s: 3.4.0 Affects Version/s: 3.2.1 > Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml > - > > Key: HDFS-15462 > URL: https://issues.apache.org/jira/browse/HDFS-15462 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > HDFS-15394 added the existing impls in core-default.xml except ofs. Let's add > ofs to core-default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15462) Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15462: -- Status: Patch Available (was: Open) > Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml > - > > Key: HDFS-15462 > URL: https://issues.apache.org/jira/browse/HDFS-15462 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > HDFS-15394 added the existing impls in core-default.xml except ofs. Let's add > ofs to core-default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15462) Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml
Siyao Meng created HDFS-15462: - Summary: Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml Key: HDFS-15462 URL: https://issues.apache.org/jira/browse/HDFS-15462 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Siyao Meng Assignee: Siyao Meng HDFS-15394 added the existing impls in core-default.xml except ofs. Let's add ofs to core-default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14920) Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission
[
https://issues.apache.org/jira/browse/HDFS-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14920:
--
Description:
Decommission test hangs in our clusters.
Have seen the messages as follow
{quote}
2019-10-22 15:58:51,514 TRACE
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block
blk_-9223372035600425840_372987973 numExpected=9, numLive=5
2019-10-22 15:58:51,514 INFO BlockStateChange: Block:
blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5,
corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4,
maintenance replicas: 0, live entering maintenance replicas: 0, excess
replicas: 0, Is Open File: false, Datanodes having this block:
10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010
10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010
10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current datanode
decommissioning: true, Is current datanode entering maintenance: false
2019-10-22 15:58:51,514 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node
10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate to
finish Decommission In Progress
{quote}
After digging the source code and cluster log, guess it happens as follow
steps.
# Storage strategy is RS-6-3-1024k.
# EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8. b0 is from
datanode dn0, b1 is from datanode dn1, ...etc
# At the beginning dn0 is in decommission progress, b0 is replicated
successfully, and dn0 is still in decommission progress.
# Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of
service, so need to reconstruct, and create ErasureCodingWork to do it, in the
ErasureCodingWork, additionalReplRequired is 4
# Because hasAllInternalBlocks is false, Will call
ErasureCodingWork#addTaskToDatanode ->
DatanodeDescriptor#addBlockToBeErasureCoded, and send BlockECReconstructionInfo
task to Datanode
# DataNode can not reconstruction the block because targets is 4, greater than
3( parity number).
There is a problem as follow, from BlockManager.java#scheduleReconstruction
{code}
// should reconstruct all the internal blocks before scheduling
// replication task for decommissioning node(s).
if (additionalReplRequired - numReplicas.decommissioning() -
numReplicas.liveEnteringMaintenanceReplicas() > 0) {
additionalReplRequired = additionalReplRequired -
numReplicas.decommissioning() -
numReplicas.liveEnteringMaintenanceReplicas();
}
{code}
Should reconstruction firstly and then replicate for decommissioning. Because
numReplicas.decommissioning() is 4, and additionalReplRequired is 4, that's
wrong,
numReplicas.decommissioning() should be 3, it should exclude live replica.
If so, additionalReplRequired will be 1, reconstruction will schedule as
expected. After that, decommission goes on.
was:
Decommission test hangs in our clusters.
Have seen the messages as follow
{quote}
2019-10-22 15:58:51,514 TRACE
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block
blk_-9223372035600425840_372987973 numExpected=9, numLive=5
2019-10-22 15:58:51,514 INFO BlockStateChange: Block:
blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5,
corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4,
maintenance replicas: 0, live entering maintenance replicas: 0, excess
replicas: 0, Is Open File: false, Datanodes having this block:
10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010
10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010
10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current datanode
decommissioning: true, Is current datanode entering maintenance: false
2019-10-22 15:58:51,514 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node
10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate to
finish Decommission In Progress
{quote}
After digging the source code and cluster log, guess it happens as follow
steps.
# Storage strategy is RS-6-3-1024k.
# EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8, b0 is from
datanode dn0, b1 is from datanode dn1, ...etc
# At the beginning dn0 is in decommission progress, b0 is replicated
successfully, and dn0 is staill in decommission progress.
# Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of
service, so need to reconstruct, and create ErasureCodingWork to do it, in the
ErasureCodingWork, additionalReplRequired is 4
# Because hasAllInternalBlocks is false, Will call
ErasureCodingWork#addTaskToDatanode ->
DatanodeDescriptor#addBlockToBeErasureCoded, and send BlockECReconstructionInfo
task to Datanode
# DataNode can not reconstruction
[jira] [Commented] (HDFS-15236) Upgrade googletest to the latest version
[
https://issues.apache.org/jira/browse/HDFS-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114321#comment-17114321
]
Siyao Meng commented on HDFS-15236:
---
No worries. Glad to help. Reinstalling protobuf 3.7.1 might solve the issue. It
is possible the protobuf header include path
({{/usr/local/include/google/protobuf/}} by default) went missing or
unregistered after updates. I used the one in BUILDING.txt:
{code:title=BUILDING.txt}
* Protocol Buffers 3.7.1 (required to build native code)
$ mkdir -p /opt/protobuf-3.7-src \
&& curl -L -s -S \
https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
\
-o /opt/protobuf-3.7.1.tar.gz \
&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C
/opt/protobuf-3.7-src \
&& cd /opt/protobuf-3.7-src \
&& ./configure\
&& make install \
&& rm -rf /opt/protobuf-3.7-src
{code}
> Upgrade googletest to the latest version
>
>
> Key: HDFS-15236
> URL: https://issues.apache.org/jira/browse/HDFS-15236
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: native, test
>Reporter: Akira Ajisaka
>Priority: Major
>
> Now libhdfspp is using gmock-1.7.0 with the patch in HDFS-15232. gmock was
> moved to googletest and the latest version is 1.10.0. Let's upgrade it to
> remove our own patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15236) Upgrade googletest to the latest version
[
https://issues.apache.org/jira/browse/HDFS-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113645#comment-17113645
]
Siyao Meng commented on HDFS-15236:
---
[~aajisaka] I am able to compile with {{mvn install -Pnative -DskipTests -e}}
on Ubuntu 20.04 LTS with OpenJDK 11.0.7.
{code:bash}
$ protoc --version
libprotoc 3.7.1
$ mvn -version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.7, vendor: Ubuntu, runtime:
/usr/lib/jvm/java-11-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-31-generic", arch: "amd64", family: "unix"
$ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn install -Pnative -DskipTests
-e
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 07:53 min
[INFO] Finished at: 2020-05-21T17:09:52-07:00
[INFO]
{code}
> Upgrade googletest to the latest version
>
>
> Key: HDFS-15236
> URL: https://issues.apache.org/jira/browse/HDFS-15236
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: native, test
>Reporter: Akira Ajisaka
>Priority: Major
>
> Now libhdfspp is using gmock-1.7.0 with the patch in HDFS-15232. gmock was
> moved to googletest and the latest version is 1.10.0. Let's upgrade it to
> remove our own patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15262) Add fsck servlet to Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15262: -- Description: Allows for running fsck on Secondary NameNode. The diff is one-line: https://github.com/smengcl/hadoop/commit/0e45887d264258b91d53c09739c182ab02cb23bb was: Allows for running fsck on Secondary NameNode. https://github.com/smengcl/hadoop/commit/0e45887d264258b91d53c09739c182ab02cb23bb > Add fsck servlet to Secondary NameNode > -- > > Key: HDFS-15262 > URL: https://issues.apache.org/jira/browse/HDFS-15262 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > > Allows for running fsck on Secondary NameNode. > The diff is one-line: > https://github.com/smengcl/hadoop/commit/0e45887d264258b91d53c09739c182ab02cb23bb -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15262) Add fsck servlet to Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15262: -- Description: Allows for running fsck on Secondary NameNode. https://github.com/smengcl/hadoop/commit/0e45887d264258b91d53c09739c182ab02cb23bb was:Allows for running fsck on Secondary NameNode. > Add fsck servlet to Secondary NameNode > -- > > Key: HDFS-15262 > URL: https://issues.apache.org/jira/browse/HDFS-15262 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > > Allows for running fsck on Secondary NameNode. > https://github.com/smengcl/hadoop/commit/0e45887d264258b91d53c09739c182ab02cb23bb -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15262) Add fsck servlet to Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng resolved HDFS-15262. --- Resolution: Won't Fix After a second thought it doesn't really make sense to put fsck in Secondary NameNode if it doesn't receive block reports from DNs. > Add fsck servlet to Secondary NameNode > -- > > Key: HDFS-15262 > URL: https://issues.apache.org/jira/browse/HDFS-15262 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > > Allows for running fsck on Secondary NameNode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15262) Add fsck servlet to Secondary NameNode
Siyao Meng created HDFS-15262: - Summary: Add fsck servlet to Secondary NameNode Key: HDFS-15262 URL: https://issues.apache.org/jira/browse/HDFS-15262 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Siyao Meng Assignee: Siyao Meng Allows for running fsck on Secondary NameNode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15262) Add fsck servlet to Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-15262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15262: -- Priority: Minor (was: Major) > Add fsck servlet to Secondary NameNode > -- > > Key: HDFS-15262 > URL: https://issues.apache.org/jira/browse/HDFS-15262 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > > Allows for running fsck on Secondary NameNode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14978) In-place Erasure Coding Conversion
[
https://issues.apache.org/jira/browse/HDFS-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14978:
--
Target Version/s: 3.4.0
> In-place Erasure Coding Conversion
> --
>
> Key: HDFS-14978
> URL: https://issues.apache.org/jira/browse/HDFS-14978
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Aravindan Vijayan
>Priority: Major
> Attachments: In-place Erasure Coding Conversion.pdf
>
>
> HDFS Erasure Coding is a new feature added in Apache Hadoop 3.0. It uses
> encoding algorithms to reduce disk space usage while retaining redundancy
> necessary for data recovery. It was a huge amount of work but it is just
> getting adopted after almost 2 years.
> One usability problem that’s blocking users from adopting HDFS Erasure Coding
> is that existing replicated files have to be copied to an EC-enabled
> directory explicitly. Renaming a file/directory to an EC-enabled directory
> does not automatically convert the blocks. Therefore users typically perform
> the following steps to erasure-code existing files:
> {noformat}
> Create $tmp directory, set EC policy at it
> Distcp $src to $tmp
> Delete $src (rm -rf $src)
> mv $tmp $src
> {noformat}
> There are several reasons why this is not popular:
> * Complex. The process involves several steps: distcp data to a temporary
> destination; delete source file; move destination to the source path.
> * Availability: there is a short period where nothing exists at the source
> path, and jobs may fail unexpectedly.
> * Overhead. During the copy phase, there is a point in time where all of
> source and destination files exist at the same time, exhausting disk space.
> * Not snapshot-friendly. If a snapshot is taken prior to performing the
> conversion, the source (replicated) files will be preserved in the cluster
> too. Therefore, the conversion actually increase storage space usage.
> * Not management-friendly. This approach changes file inode number,
> modification time and access time. Erasure coded files are supposed to store
> cold data, but this conversion makes data “hot” again.
> * Bulky. It’s either all or nothing. The directory may be partially erasure
> coded, but this approach simply erasure code everything again.
> To ease data management, we should offer a utility tool to convert replicated
> files to erasure coded files in-place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13916) Distcp SnapshotDiff to support WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13916: -- Target Version/s: 3.3.1, 3.4.0 > Distcp SnapshotDiff to support WebHDFS > -- > > Key: HDFS-13916 > URL: https://issues.apache.org/jira/browse/HDFS-13916 > Project: Hadoop HDFS > Issue Type: New Feature > Components: distcp, webhdfs >Affects Versions: 3.0.1, 3.1.1 >Reporter: Xun REN >Assignee: Xun REN >Priority: Major > Labels: easyfix, newbie, patch > Attachments: HDFS-13916.002.patch, HDFS-13916.003.patch, > HDFS-13916.004.patch, HDFS-13916.005.patch, HDFS-13916.006.patch, > HDFS-13916.patch > > > [~ljain] has worked on the JIRA: HDFS-13052 to provide the possibility to > make DistCP of SnapshotDiff with WebHDFSFileSystem. However, in the patch, > there is no modification for the real java class which is used by launching > the command "hadoop distcp ..." > > You can check in the latest version here: > [https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java#L96-L100] > In the method "preSyncCheck" of the class "DistCpSync", we still check if the > file system is DFS. > So I propose to change the class DistCpSync in order to take into > consideration what was committed by Lokesh Jain. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1987) Fix listStatus API
[ https://issues.apache.org/jira/browse/HDDS-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-1987: - Status: Patch Available (was: In Progress) > Fix listStatus API > -- > > Key: HDDS-1987 > URL: https://issues.apache.org/jira/browse/HDDS-1987 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to fix listStatus API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response. It will be picked by double buffer thread and > flushed to disk later. So when user call listStatus, it should use both > in-memory cache and rocksdb key table to return the correct result. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2455) Implement MiniOzoneHAClusterImpl#getOMLeader
[ https://issues.apache.org/jira/browse/HDDS-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-2455: - Status: Patch Available (was: In Progress) > Implement MiniOzoneHAClusterImpl#getOMLeader > > > Key: HDDS-2455 > URL: https://issues.apache.org/jira/browse/HDDS-2455 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Implement MiniOzoneHAClusterImpl#getOMLeader and use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2105) Merge OzoneClientFactory#getRpcClient functions
[ https://issues.apache.org/jira/browse/HDDS-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDDS-2105: - Status: Patch Available (was: In Progress) > Merge OzoneClientFactory#getRpcClient functions > --- > > Key: HDDS-2105 > URL: https://issues.apache.org/jira/browse/HDDS-2105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ref: https://github.com/apache/hadoop/pull/1360#discussion_r321585214 > There will be 5 overloaded OzoneClientFactory#getRpcClient functions (when > HDDS-2007 is committed). They contains some redundant logic and unnecessarily > increases code paths. > Goal: Merge those functions into fewer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
