[
https://issues.apache.org/jira/browse/HDFS-15614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320757#comment-17320757
]
Ayush Saxena commented on HDFS-15614:
-------------------------------------
[~shashikant], You tried through shell, I told it creates trash directory when
executed through HDFSAdmin. :(
See here above:
{quote}Secondly, An ambiguity, A client did an allowSnapshot say not from
HdfsAdmin he didn't had any Trash directory in the snapshot dir,
{quote}
But ok, Let me show a case where you can see some stuff through shell also, I
deployed a single node cluster on top of your PR-2881 and through shell:
{noformat}
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -mkdir /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -put
bin/yarn /dir/file1
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-setQuota 1 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-allowSnapshot /dir
allowSnapshot: The NameSpace quota (directories and files) of directory /dir is
exceeded: quota=1 file count=3
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyNamespaceQuota(DirectoryWithQuotaFeature.java:188)
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:221)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1230)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1061)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:1378)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir(FSDirMkdirOp.java:225)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createSingleDirectory(FSDirMkdirOp.java:169)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:77)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3483)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1156)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:746)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:537)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1037)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2972)
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfsadmin
-setQuota 2 /dir
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs
lsSnapshottableDir
drwxr-xr-x 0 ayushsaxena staff 0 2021-04-14 12:26 0 65536 /dir
//Namenode is working as of now, lets restart!!!
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % sbin/hadoop-daemon.sh
stop namenode
WARNING: Use of this script to stop HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon stop" instead.
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % sbin/hadoop-daemon.sh
start namenode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % jps
88755
36531 DataNode
37158 Jps
// No Namenode
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -ls /
2021-04-14 12:29:00,026 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ls: Call From ayushsaxena-MBP16.local/192.168.0.194 to localhost:9000 failed on
connection exception: java.net.ConnectException: Connection refused; For more
details see: http://wiki.apache.org/hadoop/ConnectionRefused
// Lets check the logs....
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % tail -f
logs/hadoop-ayushsaxena-namenode-ayushsaxena-MBP16.local.log
2021-04-14 12:28:51,113 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: initializing
replication queues
2021-04-14 12:28:51,113 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving
safe mode after 0 secs
2021-04-14 12:28:51,114 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network
topology has 0 racks and 0 datanodes
2021-04-14 12:28:51,114 INFO org.apache.hadoop.hdfs.StateChange: STATE*
UnderReplicatedBlocks has 0 blocks
2021-04-14 12:28:51,144 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Trash doesn't exist for
snapshottable directory /dir/. Creating trash at /dir/.Trash
2021-04-14 12:28:51,145 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: Could not provision Trash directory for existing snapshottable
directories. Exiting Namenode.
2021-04-14 12:28:51,146 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ayushsaxena-MBP16.local/XXXXXXXXX
************************************************************/
^C
ayushsaxena@ayushsaxena-MBP16 hadoop-3.4.0-SNAPSHOT % bin/hadoop version
Hadoop 3.4.0-SNAPSHOT
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r
1e53a4942ff8ae8055981a19d796303ad2b7a2b5
Compiled by ayushsaxena on 2021-04-14T06:36Z
Compiled with protoc 3.7.1
>From source with checksum 603032acda24b60f1a64568c4fc128
This command was run using
/Users/ayushsaxena/code/hadoop-code/osCode/hadoop/hadoop-dist/target/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.4.0-SNAPSHOT.jar
{noformat}
Just a point to mention, most of the calls won't be coming through shell, they
found be shooting FileSystem Api's only. That is why I tried through Fs calls,
This is just one exploit through quota, because that is easy, there are bunch
of cases where this can happen, say all cases where mkdir can fail.
I will put up a patch containing all these tests in for UT's for you to try
sometime late today/tomorrow .
> Initialize snapshot trash root during NameNode startup if enabled
> -----------------------------------------------------------------
>
> Key: HDFS-15614
> URL: https://issues.apache.org/jira/browse/HDFS-15614
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> This is a follow-up to HDFS-15607.
> Goal:
> Initialize (create) snapshot trash root for all existing snapshottable
> directories if {{dfs.namenode.snapshot.trashroot.enabled}} is set to
> {{true}}. So admins won't have to run {{dfsadmin -provisionTrash}} manually
> on all those existing snapshottable directories.
> The change is expected to land in {{FSNamesystem}}.
> Discussion:
> 1. Currently in HDFS-15607, the snapshot trash root creation logic is on the
> client side. But in order for NN to create it at startup, the logic must
> (also) be implemented on the server side as well. -- which is also a
> requirement by WebHDFS (HDFS-15612).
> 2. Alternatively, we can provide an extra parameter to the
> {{-provisionTrash}} command like: {{dfsadmin -provisionTrash -all}} to
> initialize/provision trash root on all existing snapshottable dirs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
