[
https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298823#comment-16298823
]
Steve Loughran commented on HADOOP-14630:
-----------------------------------------
TestLocalFSContractRename failing
{code}
common/target/test/data/2/fEJaHzwpnD/testRenameFileUnderFileSubdir/file
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:553)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:303)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:292)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:393)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.rename(AbstractFSContractTestBase.java:372)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContractRenameTest.java:327)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileUnderFileSubdir(AbstractContractRenameTest.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
and
{code}
Stacktrace
org.apache.hadoop.fs.FileAlreadyExistsException: Destination exists and is not
a directory:
/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/cE76lDnwXc/testRenameFileUnderFile/file
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:559)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:534)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:303)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:292)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:393)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
at
org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
at
org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:617)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.rename(AbstractFSContractTestBase.java:372)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContractRenameTest.java:327)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileUnderFile(AbstractContractRenameTest.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
This is amusing. The operation is calling {{srcFile.renameTo(dest)}}, which
returns false. so then falls back to a copy, which fails in the mkdir() phase.
Conclusion: trying to rename somethng under a file raises an exception.
I think we should just document this as a special case, make it an option &
have the ADL and native ones do it, any others if they want to.
> Contract Tests to verify create, mkdirs and rename under a file is forbidden
> ----------------------------------------------------------------------------
>
> Key: HADOOP-14630
> URL: https://issues.apache.org/jira/browse/HADOOP-14630
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, fs/swift
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch,
> HADOOP-14630-003.patch
>
>
> Object stores can get into trouble in ways which an FS would never, do, ways
> so obvious we've never done tests for them. We know what the problems are:
> test for file and dir creation directly/indirectly under other files
> * mkdir(file/file)
> * mkdir(file/subdir)
> * dir under file/subdir/subdir
> * dir/dir2/file, verify dir & dir2 exist
> * dir/dir2/dir3, verify dir & dir2 exist
> * rename(src, file/dest)
> * rename(src, file/dir/dest)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
