[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153394#comment-15153394
]
Chris Nauroth commented on HADOOP-12535:
----------------------------------------
Hi [~madhuch-ms]. It's great to see these tests being added! Thank you. It
looks like the contract tests have already revealed a handful of subtle bugs
too.
Here are some comments on the v002 patch.
# The "Contract" sub-directory should be renamed to "contract" (lower-case).
# In wasb.xml, I recommend using XInclude to include azure-auth-keys.xml. The
definition of {{fs.contract.test.fs.wasb}} would then move to
azure-auth-keys.xml. Take a look at azure-test.xml for an existing example of
how to do this. azure-auth-keys.xml is a separate external file for holding a
developer's Azure Storage credentials. It is ignored by revision control
(listed in .gitignore at the root of the repository). This has the benefit
that developers doing testing don't need to modify a file that is under
revision control, and there is less risk that someone would accidentally commit
private credentials to the repository.
# The patch includes a {{TestAzureNativeContractRootDir}}, however the tests
end up getting skipped because {{fs.contract.test.root-tests-enabled}} is set
to {{false}}. I tried enabling the tests, and there were some failures. I
didn't dig deeper to find out if this means there is a bug or if root folder
access just isn't supported fully with WASB. Could you please take a look? If
it turns out that root folder access isn't fully supported, then I suggest
deleting {{TestAzureNativeContractRootDir}}.
# HADOOP-12635 added {{append}} support for WASB, so we should be able to set
{{fs.contract.supports-append}} to {{true}} and add a
{{TestNativeAzureContractAppend}} suite. I tried doing this and I saw one
failure in {{AbstractContractAppendTest#testRenameFileBeingAppended}}. Maybe
[~dchickabasapa] could help investigate if this is a bug?
# It seems there ought to be a way to use the optional WASB support for atomic
rename and enable {{fs.contract.supports-atomic-directory-delete}} and
{{fs.contract.supports-atomic-rename}}. Have you investigated this?
# Let's also try enabling {{fs.contract.supports-getfilestatus}} and adding a
{{TestAzureNativeContractGetFileStatus}}, which subclasses
{{AbstractContractGetFileStatusTest}}.
# There are a few additional contract options. For the sake of completion,
let's list all of these in wasb.xml: {{fs.contract.rename-creates-dest-dirs}},
{{fs.contract.rename-returns-false-if-dest-exists}},
{{fs.contract.rename-remove-dest-if-empty-dir}},
{{fs.contract.supports-settimes}}, {{fs.contract.supports-getfilestatus}} and
{{fs.contract.supports-available-on-closed-file}}. The HDFS contract is at
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/contract/hdfs.xml. The
closer we can get to matching that contract, the closer we are to a fully
compatible alternative file system implementation.
# Let's also update the documentation at src/site/markdown/index.md with a
mention of the contract tests, and update the example azure-auth-keys.xml file
as I described above.
> Run FileSystem contract tests with hadoop-azure.
> ------------------------------------------------
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
> Reporter: Chris Nauroth
> Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
