ConfX created HDFS-17104:
----------------------------
Summary: permission deprivation in HDFS without proper check or
restoration
Key: HDFS-17104
URL: https://issues.apache.org/jira/browse/HDFS-17104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened?
After setting {{dfs.namenode.storage.dir.perm}} to {{{}325{}}}, running
{{org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot#testParentDirWithUCFileDeleteWithSnapShot}}
creates a folder that cannot be deleted by the current user unless using
{{{}sudo{}}}.
h2. Where's the bug?
Many of the storage systems in HDFS, e.g., {{NNStorage}} and {{{}JNStorage{}}},
uses {{Storage.clearDirectory}} as a way to create a fresh folder for the
storage. In this method after creating the directory the storage attempts to
changes the permission
{noformat}
if (permission != null) {
try {
Set<PosixFilePermission> permissions =
PosixFilePermissions.fromString(permission.toString());
Files.setPosixFilePermissions(curDir.toPath(), permissions);
} catch (UnsupportedOperationException uoe) {
// Default to FileUtil for non posix file systems
FileUtil.setPermission(curDir, permission);
}
}{noformat}
However, if the {{permission}} is set such that the current user does not have
certain permissions, the newly created folder would because very trick to deal
with since both the Hadoop and the current user cannot modify and/or read it.
h2. How to reproduce?
(1) Set {{dfs.namenode.storage.dir.perm}} to {{325}}
(2) Run
{{org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot#testParentDirWithUCFileDeleteWithSnapShot}}
(3) Checks
{{/Users/tonywang/confuzz-project/fuzz-hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs}}
One could also run
{{org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot}}
to observe that the exception being thrown changes from
{{FileNotFoundException}} to {{{}IOException{}}}.
h2. Stacktrace
The first stacktrace:
{noformat}
java.io.FileNotFoundException: No valid image files found
at
org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector.getLatestImages(FSImageTransactionalStorageInspector.java:158)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:688)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:339)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1201)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:768)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1020)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:995)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1769)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1374)
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1143)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1016)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:948)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:576)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.setup(TestOpenFilesWithSnapshot.java:70){noformat}
The second stacktrace:
{noformat}
java.io.IOException: Could not fully delete
/Users/tonywang/confuzz-project/fuzz-hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name-0-1
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1097)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1016)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:948)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:576)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.setup(TestOpenFilesWithSnapshot.java:70){noformat}
For an easy reproduction, run the reproduce.sh in the attachment. We are happy
to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
