[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325352#comment-17325352
]
Arpit Agarwal commented on HDFS-15614:
--
bq. , if providing an external command to create the Trash directory by admins
is feasible and makes sense
The external command will add more friction to enabling the feature. We want it
to be transparent as far as possible. I like the option to auto create the
.Trash dir better.
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320982#comment-17320982
]
Ayush Saxena commented on HDFS-15614:
-
Quoting [~shashikant] from the
[PR|https://github.com/apache/hadoop/pull/2881#issuecomment-819454476]
{quote}Coming to think of it, if providing an external command to create the
Trash directory by admins is feasible and makes sense, i think its ok to remove
the NN startup logic to create Trash directories inside snapshottable root.
{quote}
Yeps, thats exactly my point, We seems to be on same page now. :)
[~smeng]
{quote}Therefore this would be the same deal for encryption zone trash root
creation. When replacing dfs.allowSnapshot calls with dfs.createEncryptionZone
in the first test case we should also find trash root inside encryption zone
missing with dfs.provisionSnapshotTrash call alone.
{quote}
Well I didn't actually catch this, but the Ambiguity I am talking about is not
between commands executed from DFSAdmin and DFS, The ambiguity I am talking
about is the change coming post namenode startup/failover. If it was there
before failover/restart. It should be there post failover/restart. If it wasn't
there before so it shouldn't surface also after failover/restart.
EncryptionZones doesn't create any trash dir themselves during restart or
failover AFAIK. So, this ambiguity will not be there. And the existence of
Trash in case of DFSAdmin and not through DFS, is OK for both yours and
Encryption zone cases. That is just a behavioural aspect.
{quote}name quota can become an issue indeed.
{quote}
{quote}name quota can become an issue indeed.
I think I got your point. Maybe a better way to create those necessary Trash
dirs is to ask an admin to run dfsadmin command manually after flipping
dfs.namenode.snapshot.trashroot.enabled to true.
Currently we already have dfsadmin -provisionSnapshotTrash but can only be
done one by one. dfsadmin -provisionSnapshotTrash -all can be implemented to
achieve this.
{quote}
Yeps, We have an agreement again. :) This way sounds very appropriate. Having
{{dfsadmin -provisionSnapshotTrash -all}} will save efforts and will be really
good, Should be very doable as well at client side itself. Can go to Namenode
too, but going to the namenode to create for all in one shot might lead to lock
starvation and stuff.
So, If everyone is on same page(I suppose). We can get rid of HDFS-15614 and
HDFS-15820 and if any other fix on top. and open up a new jira for -all stuff.
Thanx Shashikant and Siyao!!!
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320927#comment-17320927
]
Siyao Meng commented on HDFS-15614:
---
Thanks for bringing this up [~ayushtkn].
{quote}
And this fails, And yep there is an ambiguity.
{quote}
The reason is that
[{{DFS#provisionSnapshotTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2984]
followed EZ counterpart's
[{{DFS#provisionEZTrash}}|https://github.com/apache/hadoop/blob/c6539e3289711d29f508930bbda40302f48ddf4c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2913]
implementation. {{dfs.provisionSnapshotTrash}} is not automatically called
from {{dfs.allowSnapshot}}, following what .
Therefore this would be the same deal for encryption zone trash root creation.
When replacing {{dfs.allowSnapshot}} calls with {{dfs.createEncryptionZone}} in
the first test case we should also find trash root inside encryption zone
missing with {{dfs.provisionSnapshotTrash}} call *alone*.
I suggest some guidelines should be posted and in javadoc adding that
allowSnapshot should better performed with dfsadmin CLI (and
createEncryptionZone if there aren't already).
{quote}
How come a client side feature that important, that can make the cluster go
down in times of critical situation like failover, Again a test to show that:
{quote}
[name
quota|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Name_Quotas]
can become an issue indeed.
I think I got your point. Maybe a better way to create those necessary Trash
dirs is to ask an admin to run dfsadmin command *manually* after flipping
{{dfs.namenode.snapshot.trashroot.enabled}} to {{true}}.
Currently we already have {{dfsadmin -provisionSnapshotTrash}} but can only be
done one by one. {{dfsadmin -provisionSnapshotTrash -all}} can be implemented
to achieve this.
Cheers,
Siyao
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320757#comment-17320757
]
Ayush Saxena commented on HDFS-15614:
-
[~shashikant], You tried through shell, I told it creates trash directory when
executed through HDFSAdmin. :(
See here above:
{quote}Secondly, An ambiguity, A client did an allowSnapshot say not from
HdfsAdmin he didn't had any Trash directory in the snapshot dir,
{quote}
But ok, Let me show a case where you can see some stuff through shell also, I
deployed a single node cluster on top of your PR-2881 and through shell:
{noformat}
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -mkdir /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -put
bin/yarn /dir/file1
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-setQuota 1 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-allowSnapshot /dir
allowSnapshot: The NameSpace quota (directories and files) of directory /dir is
exceeded: quota=1 file count=3
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyNamespaceQuota(DirectoryWithQuotaFeature.java:188)
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:221)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1230)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1061)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:1378)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir(FSDirMkdirOp.java:225)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createSingleDirectory(FSDirMkdirOp.java:169)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:77)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3483)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1156)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:746)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:537)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1037)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2972)
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-setQuota 2 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs
lsSnapshottableDir
drwxr-xr-x 0 ayushsaxena staff 0 2021-04-14 12:26 0 65536 /dir
//Namenode is working as of now, lets restart!!!
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % sbin/hadoop-daemon.sh
stop namenode
WARNING: Use of this script to stop HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon stop" instead.
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % sbin/hadoop-daemon.sh
start namenode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % jps
88755
36531 DataNode
37158 Jps
// No Namenode
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -ls /
2021-04-14 12:29:00,026 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ls: Call From ayushsaxena-MBP16.local/192.168.0.194 to localhost:9000 failed on
connection exception: java.net.ConnectException: Connection refused; For more
details see: http://wiki.apache.org/hadoop/ConnectionRefused
// Lets check the logs
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % tail -f
logs/hadoop-ayushsaxena-namenode-ayushsaxena-MBP16.local.log
2021-04-14 12:28:51,113 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: initializing
replication queues
2021-04-14 12:28:51,113 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving
safe mode after 0 secs
2021-04-14 12:28:51,114 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network
topology has 0 racks and 0 datanodes
2021-04-14 12:28:51,114 INFO
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320719#comment-17320719
]
Shashikant Banerjee commented on HDFS-15614:
[~ayushtkn], i just tried making a snapshottable directory and it seems the
.Trash is implicitly created once the config
"dfs.namenode.snapshot.trashroot.enabled" is set to true.
{code:java}
hdfs dfsadmin -fs hdfs://127.0.0.1: -allowsnapshot /
hdfs dfs -ls hdfs://127.0.0.1:/
Found 2 items
drwxrwxrwt - shashikant supergroup 0 2021-04-12 11:20
hdfs://127.0.0.1:/.Trash
drwxr-xr-x - shashikant supergroup 0 2021-04-12 11:19
hdfs://127.0.0.1:/dir1{code}
[~smeng], can you please confirm?
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319274#comment-17319274
]
Ayush Saxena commented on HDFS-15614:
-
Thanx [~shashikant] for the responses, I will bother you a bit more:
{quote}If a new directory is made snapshottable with feature flahg turned ,
.Trash directory gets created implicitly as a part of allowSnapshot call. I
don't think there is an ambiguity here.
{quote}
I think I cleared this up in my question itself, So here is a test for that:
{code:java}
@Test
public void testClientAmbiguity() throws Exception {
Configuration conf = new HdfsConfiguration();
// Enable the feature
conf.setBoolean("dfs.namenode.snapshot.trashroot.enabled", true);
try (MiniDFSCluster cluster =
new MiniDFSCluster.Builder(conf).build()) {
cluster.waitActive();
final DistributedFileSystem dfs = cluster.getFileSystem();
// Create two directories, on 1 allowSnapshot through DFS & on other
// from HDFS ADMIN
Path dir1 = new Path("/dir1");
Path dir2 = new Path("/dir2");
dfs.mkdirs(dir1);
dfs.mkdirs(dir2);
// AllowSnapshot on dir1 through dfs
dfs.allowSnapshot(dir1);
// AllowSnapshot on dir2 through dfsadmin
DFSAdmin dfsAdmin = new DFSAdmin(conf);
ToolRunner.run(conf, dfsAdmin,new String[]{"-allowSnapshot",
dir2.toString()});
// Check for trash directory in dir1(allowed through dfs)
assertFalse(dfs.exists(new Path(dir1,FileSystem.TRASH_PREFIX))); // (1)
// Check for trash directory in dir2(allowed through DfsAdmin)
assertTrue(dfs.exists(new Path(dir2,FileSystem.TRASH_PREFIX)));
// Failover/Restart namenode and stuff
cluster.restartNameNodes();
cluster.waitActive();
// Nothing should change
// Check for trash directory in dir1(allowed through dfs)
// Will fail here. stuff changed post restart of namenode. Such thing
// will happen with uupgrade as well.
assertFalse(dfs.exists(new Path(dir1,FileSystem.TRASH_PREFIX))); // (1)
assertTrue(dfs.exists(new Path(dir2,FileSystem.TRASH_PREFIX)));
}
}
{code}
And this fails, And yep there is an ambiguity.
{quote}This is important for provisioning snapshot trash to use ordered
snapshot deletion feature if the system already had pre existing snapshottable
directories.
{quote}
How come a client side feature that important, that can make the cluster go
down in times of critical situation like failover, Again a test to show that:
{code:java}
@Test
public void testFailureAfterFailoverOrRestart() throws Exception {
Configuration conf = new HdfsConfiguration();
// Enable the feature
conf.setBoolean("dfs.namenode.snapshot.trashroot.enabled", true);
try (MiniDFSCluster cluster =
new MiniDFSCluster.Builder(conf).build()) {
cluster.waitActive();
final DistributedFileSystem dfs = cluster.getFileSystem();
// Create a directory
Path dir1 = new Path("/dir1");
dfs.mkdirs(dir1);
// AllowSnapshot on dir1
dfs.allowSnapshot(dir1);
// SetQuota
dfs.setQuota(dir1, 1, 1);
// Check if the cluster is working and happy.
dfs.mkdirs(new Path("/dir2"));
assertTrue(dfs.exists(new Path("/dir2")));
// Failover/Restart namenode or such stuff
cluster.restartNameNodes(); // Namenode Crashed, It was failover, then
// standby would also crash & ultimately whole of cluster.
// Will not reach here itself. :-(
cluster.waitActive();
dfs.listStatus(new Path("/dir1"));
}
}
{code}
{quote}The "getAllSnapshottableDirs()" in itslef is not a heavy call IMO. It
does not depend on the no of snapshots present in the system.
{quote}
Ok If you say, getAllSnapshottableDirs() might not be heavy, even if there are
tons of snapshottable directories, So, getFileInfo for all these directories
and then mkdirs for all these in worst case.
So a normal scenario is like:
1 call getAllSnapshottableDirs -> say fetches 2 million dirs
2 million getFileInfo() calls -> say avg case 1 million doesn't have trash
1 million mkdirs() -> a write call isn't considered cheap & fast you go to JNs
and stuff.
If you get this creation of snapshots in the filesystem spec as well, still you
won't get rid of any of these problems,
Nevertheless, what ever be the case a normal running cluster shouldn't crash
due to any feature, and that too during failovers, that is some crazy stuff.
And regarding encryption zone stuff are you talking it is similar to
HDFS-10324(I see this only linked on HDFS-15607)? Well I don't think it is
doing create like stuff during startup. Will see if [~weichiu] can confirm
that, He worked on it. Didn't dig in much though
Well not very sure of the use case and things here, so would leave it to you
guys. Please don't hold anything for me in the
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319168#comment-17319168
]
Shashikant Banerjee commented on HDFS-15614:
Thanks [~ayushtkn]. The "getAllSnapshottableDirs()" in itslef is not a heavy
call IMO. It does not depend on the no of snapshots present in the system.
{code:java}
1. What if the mkdirs fail? the namenode will crash, ultimately all Namenodes
will try this stuff in an attempt to become active and come out of safemode.
Hence all the namenodes will crash. Why mkdirs can fail, could be many reasons,
I can tell you one which I tried: Namespace Quotas, and yep the namenode
crashed. can be bunch of such cases
{code}
If mkdir fails to create the Trash directory , inside the snapshot root, then
strict ordering/processing of all entries during snapshot deletion can not be
guaranteed, If this feature needs to be used, .Trash needs to be within the
snapshottable directory which is similar to the case with encryption zones.
{code:java}
2. Secondly, An ambiguity, A client did an allowSnapshot say not from HdfsAdmin
he didn't had any Trash directory in the snapshot dir, Suddenly a failover
happened, he would get a trash directory in its snapshot directory, Which he
never created.{code}
If a new directory is made snapshottable with feature flahg turned , .Trash
directory gets created impliclitly as a part of allowSnapshot call. I don't
think there is an ambiguity here.
{code:java}
Third, The time cost, The namenode startup or the namenode failover or let it
be coming out of safemode should be fast, They are actually contributing to
cluster down time, and here we are doing like first getSnapshottableDirs which
itself would be a heavy call if you have a lot of snapshots, then for each
directory, one by one we are doing a getFileInfo and then a mkdir, seems like
time-consuming. Not sure about the memory consumption at that point due to this
though...
{code}
I don't think getSnapshottableDirs() is a very heavey call in typical setups.
It has nothing to do with the no of snapshots that exist in the sytem.
{code:java}
Fourth, Why the namenode needs to do a client operation? It is the server. And
that too while starting up, This mkdirs from namenode to self is itself
suspicious, Bunch of namenode crashing coming up trying to become active,
trying to push same edits, Hopefully you would have taken that into account and
pretty sure such things won't occur, Namenodes won't collide even in the rarest
cases. yep and all safe with the permissions..
{code}
This is important for provisioning snapshot trash to use ordered snapshot
deletion feature if the system already had pre existing snapshottable
directories.
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15614) Initialize snapshot trash root during NameNode startup if enabled
[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17318685#comment-17318685
]
Ayush Saxena commented on HDFS-15614:
-
Hi [~smeng]/[~shashikant]
Observed this while checking HDFS-15961, So couple of doubts on this If you
guys can help.
So, IIUC this + the fixes, leads to a scenario like:
The Namenode during startup as {{Active Namenode / Leaving safemode / failover
to Active}}, *during* such critical process, does a heavy ready operation, i.e
{{getAllSnapshottableDirs()}} and *then a write operation mkdirs to itself*
which would be generating edits as well, and that too during this process not
like after.
So, couple of questions around it.
* What if the *mkdirs fail?* the namenode will crash, ultimately all Namenodes
will try this stuff in an attempt to become active and come out of safemode.
Hence all the namenodes will crash. Why mkdirs can fail, could be many reasons,
I can tell you one which I tried: Namespace Quotas, and yep the namenode
crashed. can be bunch of such cases
* Secondly, An ambiguity, A client did an {{allowSnapshot}} say not from
{{HdfsAdmin}} he didn't had any {{Trash}} directory in the snapshot dir,
Suddenly a failover happened, he would get a trash directory in its snapshot
directory, Which he never created.
* Third, The time cost, The namenode startup or the namenode failover or let
it be coming out of safemode should be fast, They are actually contributing to
cluster down time, and here we are doing like first {{getSnapshottableDirs}}
which itself would be a heavy call if you have a lot of snapshots, then for
each directory, one by one we are doing a {{getFileInfo}} and then a {{mkdir}},
seems like time-consuming. Not sure about the memory consumption at that point
due to this though...
* Fourth, Why the namenode needs to do a client operation? It is the server.
*And that too while starting up*, This {{mkdirs}} from namenode to self is
itself suspicious, Bunch of namenode crashing coming up trying to become
active, trying to push same edits, Hopefully you would have taken that into
account and pretty sure such things won't occur, Namenodes won't collide even
in the rarest cases. yep and all safe with the permissions...
Do help clarify when you find some bandwidth, I will also try to explore a
little more and discuss with some friends around. Thanx.
> Initialize snapshot trash root during NameNode startup if enabled
> -
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
