[
https://issues.apache.org/jira/browse/HDFS-15578?focusedWorklogId=485254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-485254
]
ASF GitHub Bot logged work on HDFS-15578:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Sep/20 17:04
Start Date: 16/Sep/20 17:04
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #2305:
URL: https://github.com/apache/hadoop/pull/2305#discussion_r489589949
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestViewDistributedFileSystemWithMountLinks.java
##########
@@ -61,4 +64,55 @@ public void testCreateOnRoot() throws Exception {
public void testMountLinkWithNonExistentLink() throws Exception {
testMountLinkWithNonExistentLink(false);
}
+
+ @Test
+ public void testRenameOnInternalDirWithFallback() throws Exception {
+ Configuration conf = getConf();
+ URI defaultFSURI =
+ URI.create(conf.get(CommonConfigurationKeys.FS_DEFAULT_NAME_KEY));
+ final Path hdfsTargetPath1 = new Path(defaultFSURI + "/HDFSUser");
+ final Path hdfsTargetPath2 = new Path(defaultFSURI + "/NewHDFSUser/next");
+ ViewFsTestSetup.addMountLinksToConf(defaultFSURI.getAuthority(),
+ new String[] {"/HDFSUser", "/NewHDFSUser/next"},
+ new String[] {hdfsTargetPath1.toUri().toString(),
+ hdfsTargetPath2.toUri().toString()}, conf);
+ //Making sure parent dir structure as mount points available in fallback.
+ try (DistributedFileSystem dfs = new DistributedFileSystem()) {
+ dfs.initialize(defaultFSURI, conf);
+ dfs.mkdirs(hdfsTargetPath1);
+ dfs.mkdirs(hdfsTargetPath2);
+ }
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ Path src = new Path("/newFileOnRoot");
+ Path dst = new Path("/newFileOnRoot1");
+ fs.create(src).close();
+ verifyRename(fs, src, dst);
+
+ src = new Path("/newFileOnRoot1");
+ dst = new Path("/NewHDFSUser/newFileOnRoot");
+ fs.mkdirs(dst.getParent());
+ verifyRename(fs, src, dst);
+
+ src = new Path("/NewHDFSUser/newFileOnRoot");
+ dst = new Path("/NewHDFSUser/newFileOnRoot1");
+ verifyRename(fs, src, dst);
+
+ src = new Path("/NewHDFSUser/newFileOnRoot1");
+ dst = new Path("/newFileOnRoot");
+ verifyRename(fs, src, dst);
+
+ src = new Path("/HDFSUser/newFileOnRoot1");
+ dst = new Path("/HDFSUser/newFileOnRoot");
+ fs.create(src).close();
+ verifyRename(fs, src, dst);
+ }
+ }
+
+ private void verifyRename(FileSystem fs, Path src, Path dst)
+ throws IOException {
+ fs.rename(src, dst);
+ Assert.assertFalse(fs.exists(src));
+ Assert.assertTrue(fs.exists(dst));
+ }
Review comment:
Thanx @umamaheswararao for the update.
Regarding the Case 2: When the same directory structure isn't available in
the fallback.
In `ViewFs` I think this was handled and `createParent` was explicitly made
`true` always. It would be just for rename this compulsion would be there.
Considering a mount entry like -- `/mount/sub1/sub2` --> `/nsPath`
if someone calls rename with `dst` as `/mount/sub1/renameFile` will fail,
but if he calls create `/mount/sub1/createFile` without `createParent` it would
pass and this `create` call will create the internal directory structure as
well. So, now again the user calls the same rename command, it would succeed.
Same for `mkdir` with `createParent` as `false`
This would be little intermittent behavior for the end user, one API
behaving differently.
Secondly creating the same directory structure at `fallback` just for
`rename` to work doesn't seems feasible, It would be too many empty
directories, increasing the number of inodes at NN. IIRC something like this,
to create empty directories for mount entries in case of RBF was discussed for
some issue recently, and UBER folks had concerns with inode numbers going high
due to empty directories.
I think we should explicitly take care of this in `rename` as well, May be
in non-atomic way only? Latter we might find a better way, Maybe adding one
more flag to `rename2` and argument to `rename` for `createParent` in a follow
up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 485254)
Time Spent: 1h 40m (was: 1.5h)
> Fix the rename issues with fallback fs enabled
> ----------------------------------------------
>
> Key: HDFS-15578
> URL: https://issues.apache.org/jira/browse/HDFS-15578
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, viewfsOverloadScheme
> Affects Versions: 3.4.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When we enable fallback, rename should success if the src.parent or
> dst.parent on inernalDir.
> {noformat}
> org.apache.hadoop.security.AccessControlException: InternalDir of
> ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.org.apache.hadoop.security.AccessControlException: InternalDir
> of ViewFileSystem is readonly, operation rename not permitted on path
> /newFileOnRoot.
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:95)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.readOnlyMountTable(ViewFileSystem.java:101)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:683) at
> org.apache.hadoop.hdfs.ViewDistributedFileSystem.rename(ViewDistributedFileSystem.java:533)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.verifyRename(TestViewDistributedFileSystemWithMountLinks.java:114)
> at
> org.apache.hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks.testRenameOnInternalDirWithFallback(TestViewDistributedFileSystemWithMountLinks.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
