[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159609#comment-15159609
]
madhumita chakraborty commented on HADOOP-12535:
------------------------------------------------
Took care if the comments and uploaded new patch.
[~cnauroth] Some doubts
1. How do atomic rename and delete settings
(fs.contract.supports-atomic-directory-delete and
fs.contract.supports-atomic-directory-rename) get used? I dont see any
reference to SUPPORTS_ATOMIC_RENAME and SUPPORTS_ATOMIC_DIRECTORY_DELETE
defined in ContractOptions. If a filesystem supports atomic rename, do they
need to write new contract test?
2. I added contract test for append. testRenameFileBeingAppended if
failing. I have skipped this test for now. This test case appends a file and
then renames it.
FSDataOutputStream outputStream = getFileSystem().append(target);
outputStream.write(dataset);
Path renamed = new Path(testPath,”renamed”);
Rename(target, renamed);
outputStream.close();
So here we see that after renaming it closes the original file stream. Was this
test written in this way to maintain parity with unix file system? But WASB
does not support this semantic. If the file is renamed before closing the
original stream then we get 2 types of error.
a. In append we take SelfRenewingLease on the blob. In Close we see if the
lease has expired it will try to renew the lease and that will throw exception
as the blob has already been deleted as part of rename.
b. In close we commit the blocks of blockblob. So if we rename the file
before closing the stream, actually the content does not get committed. And
empty renamed file is created.
cc [~dchickabasapa]
3.Root folder access is not fully supported in WASB. So removed
TestAzureNativeContractRootDir.
> Run FileSystem contract tests with hadoop-azure.
> ------------------------------------------------
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
> Reporter: Chris Nauroth
> Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
