[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933438#comment-15933438 ] John Zhuge commented on HADOOP-12875: - Thanks [~vishwajeet.dusane]. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs/adl, test, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Fix For: 3.0.0-alpha1 > > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933295#comment-15933295 ] Vishwajeet Dusane commented on HADOOP-12875: I think HADOOP-13257 ported to branch 2.8 should already cover the changes from this patch. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs/adl, test, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Fix For: 3.0.0-alpha1 > > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933261#comment-15933261 ] John Zhuge commented on HADOOP-12875: - [~chris.douglas], [~vishwajeet.dusane] Should we backport this to branch-2.8.0 as well? Would love pass live ADLS unit tests in all supported branches. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs/adl, test, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Fix For: 3.0.0-alpha1 > > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334144#comment-15334144 ] Hudson commented on HADOOP-12875: - SUCCESS: Integrated in Hadoop-trunk-Commit #9969 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9969/]) HADOOP-12875. [Azure Data Lake] Support for contract test and unit test (cnauroth: rev c9e71382a58b6ffcb3fccb79d3c146877f1c8313) * hadoop-tools/hadoop-azure-datalake/src/test/resources/adls.xml * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractOpenLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/AdlMockWebServer.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestSplitSizeCalculation.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageConfiguration.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractDeleteLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractConcatLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestAdlRead.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractAppendLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextCreateMkdirLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlFileSystemContractLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractRenameLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConcurrentDataReadOperations.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractMkdirLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestListStatus.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/AdlStorageContract.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlReadLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestGetFileStatus.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractRootDirLive.java * hadoop-tools/hadoop-azure-datalake/src/test/resources/contract-test-options.xml * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlWebHdfsFileContextMainOperationsLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractCreateLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/hdfs/web/TestConfigurationSetting.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestableAdlFileSystem.java * hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/hdfs/web/PrivateAzureDataLakeFileSystem.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlDifferentSizeWritesLive.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/oauth2/TestCachedRefreshTokenBasedAccessTokenProvider.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/ExpectedResponse.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestADLResponseData.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/common/TestDataForRead.java * hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/live/TestAdlContractSeekLive.java > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Fix For: 3.0.0-alpha1 > > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327336#comment-15327336
]
Vishwajeet Dusane commented on HADOOP-12875:
[~cnauroth] - White-space issue is originated from file
{./hadoop-build-tools/src/main/resources/META-INF/LICENSE.txt} and not
introduced in this patch.
For Findbug issue reported - I have removed Math.round usage as well but still
the warning has been pointed on the line which does not contain Math.Round
usage. The issue does look like to be ignored, Could you please have a look
once ?
> [Azure Data Lake] Support for contract test and unit test cases
> ---
>
> Key: HADOOP-12875
> URL: https://issues.apache.org/jira/browse/HADOOP-12875
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch,
> Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch
>
>
> This JIRA describes contract test and unit test cases support for azure data
> lake file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327154#comment-15327154
]
Hadoop QA commented on HADOOP-12875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
28s{color} | {color:red} hadoop-tools/hadoop-azure-datalake in trunk has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 20 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} hadoop-tools/hadoop-azure-datalake generated 0 new +
0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
23s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809807/Hadoop-12875-005.patch
|
| JIRA Issue | HADOOP-12875 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 1ca230bf08ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 28b66ae |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9758/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-azure-datalake-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9758/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9758/testReport/ |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9758/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327141#comment-15327141 ] Vishwajeet Dusane commented on HADOOP-12875: [~cnauroth] and [~ste...@apache.org], thanks a lot for being flexible. [~ste...@apache.org] - Existing failing cases requires some change in our Adls backend server. Since the change to reflect in production environment is not immediate as of today and requires to go through internal test cycles, we agreed to mark those test as bug and once the fix is available in production environment, we would uncomment the overridden stub to execute the test. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch, Hadoop-12875-005.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325115#comment-15325115 ] Chris Nauroth commented on HADOOP-12875: Steve, thank you for the flexibility. I have filed HADOOP-13257 and copy-pasted your comments. [~vishwajeet.dusane], the remaining thing we need to do on this patch is to address the warnings from pre-commit: # Please fix the Findbugs warning. # Please fix the Checkstyle warnings. Sometimes Checkstyle warnings are overly pedantic and we proceed without addressing all of them. Feel free to ask me if you think you're looking at a warning that might fall into that category, and I'd be happy to advise on whether or not I think it's necessary. # Pre-commit is reporting tests as failing, because it won't have ADL credentials. You probably need to adjust the test configuration setup and the pom.xml file so that tests won't get triggered if the credentials file is missing. I recommend taking a look at the pom.xml in hadoop-aws for an example of how to do that. bq. I just do think all those skipped contract tests need to get working, for the benefit of everyone. Absolutely. They are invaluable for catching issues before more costly long-running system tests. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325092#comment-15325092 ] Steve Loughran commented on HADOOP-12875: - OK, i'll lift it. Given the functional code is in there, the tests do at least track regressions. =0. I just do think all those skipped contract tests need to get working, for the benefit of everyone. I know doing it hurts: I've just managed to break listStatus() quite dramatically in a test run next to this browser and spent 2+ hours in the nuances of listFiles, but they are the closest thing we have in the hadoop codebase for trying to define what these implementations do. they are needed: and if they fail, that's a warning sign, not something to skip I'd propose a followup JIRA then "improve contract tests", taking in my concerns > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325067#comment-15325067 ] Chris Nauroth commented on HADOOP-12875: [~ste...@apache.org], building on discussion in HADOOP-12666, our plan was to commit that and this together with known follow-on work to be tracked in separate JIRAs. This would have broken the multi-month logjam on HADOOP-12666 and allowed further progress to proceed more iteratively and easily. The contract tests, while still needing the additional work you pointed out, are still going to be helpful for catching regressions during subsequent work. Can I please ask you to compromise, lift your -1, and defer these additional items to be tracked in separate JIRAs? If not, then would you at least consider deferring a subset of those items? I had kicked off a pre-commit run right after committing HADOOP-12666. [~vishwajeet.dusane], the pre-commit warnings definitely need to be addressed, including the Findbugs warning. Mysteriously, that Findbugs warning was not flagged during pre-commit of HADOOP-12666. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch, > Hadoop-12875-003.patch, Hadoop-12875-004.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324183#comment-15324183
]
Steve Loughran commented on HADOOP-12875:
-
-1
Various tests are being skipped with a "BUG" commentary. those are signs of
where the FS doesn't behave as expected. It's generally acceptable to skip
tests where the feature is unimplemented especially in the quirks of object
stores —but not where things are fundamentally at odds with [the filesystem
specification|https://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-common/filesystem/filesystem.html]
As an example, the test {{testRmNonEmptyRootDirNonRecursive()}} is skipped.
That test is verifying a core feature of the FS behaviour. Same for
{{AbstractContractMkdirTest.testNoMkdirOverFile()}}, where we say "you cannot
create a directory where there is a file".
* TestListStatus uses {{System.out}}. Replace with logging
* Contract tests must not be using {{fs.default.fs}} as source; makes it
impossible to have >1 FS in a test module. Look at how hadoop-aws defines a
separate FS URI for each FS in options like {{test.fs.s3n.name}} and
{{test.fs.s3a.name}}
* I don't understand why every setup has been overridden with a check for the
test being enabled. The {{AdlStorageContract}} class should subclass
{{isEnabled()}} for this
* Add a test for a seek of a long negative number, expect it throw an
EOFException. Your seek() bounds checking only checks for -1, which is only a
small subset of the possible negative seek ranges. This test should actually go
into {{AbstractContractSeekTest}}, as we should see what other filesystems get
up to.
* Rather than skip some of {{AbstractContractMkdirTest}}, these MUST be fixed.
* In {{TestConfigurationSetting}}}, all the {{close()}} calls must be designed
to complete even in the presence of assertion failures. Use try-with-resources.
> [Azure Data Lake] Support for contract test and unit test cases
> ---
>
> Key: HADOOP-12875
> URL: https://issues.apache.org/jira/browse/HADOOP-12875
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch,
> Hadoop-12875-003.patch, Hadoop-12875-004.patch
>
>
> This JIRA describes contract test and unit test cases support for azure data
> lake file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323628#comment-15323628
]
Hadoop QA commented on HADOOP-12875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 29 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} hadoop-tools/hadoop-azure-datalake in trunk has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hadoop-tools/hadoop-azure-datalake: The patch
generated 82 new + 0 unchanged - 0 fixed = 82 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 20 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 16s{color}
| {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.adl.live.TestAdlFileSystemContractLive |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12803230/Hadoop-12875-004.patch
|
| JIRA Issue | HADOOP-12875 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux ff48f6f32063 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9581fb7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9715/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-azure-datalake-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9715/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9715/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9715/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-datalake.txt
|
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278227#comment-15278227
]
Hadoop QA commented on HADOOP-12875:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 29 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 0s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 29s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 10s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12803230/Hadoop-12875-004.patch
|
|
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238685#comment-15238685
]
Hadoop QA commented on HADOOP-12875:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 28 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 51s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 25s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 105m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12798437/Hadoop-12875-003.patch
|
| JIRA Issue |
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237466#comment-15237466 ] Vishwajeet Dusane commented on HADOOP-12875: I will update the patch earliest with ASF license for the missing 7 newly added files. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237451#comment-15237451
]
Hadoop QA commented on HADOOP-12875:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 28 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 43m 53s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 9s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 59s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12798260/Hadoop-12875-002.patch
|
| JIRA Issue | HADOOP-12875 |
|
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237136#comment-15237136
]
Vishwajeet Dusane commented on HADOOP-12875:
[~cnauroth] - Thanks for the review.
1. For .gitignore - Should this file be added after the initial checkin ?
2. Below code is not necessary, i have removed that code.
{code}
System.setProperty("hadoop.home.dir", System.getProperty("user.dir"));
{code}
3. Benchmark test were to get latency of Adl driver alone to finds an overhead
adl driver logic is adding. I have removed those test because that should be a
different benchmark suite altogether and need not be part of this patchset.
4. using {{org.apache.hadoop.util.Time#monotonicNow}} now instead of the
{{System#currentTimeMillis}}
5. Throwing {{IllegalStateException}} instead of {{NullPointerException}}, this
would be useful for developer to debug quickly on what had gone wrong. Thanks
again for this comment.
6. Created separate class for non contract test cases.
> [Azure Data Lake] Support for contract test and unit test cases
> ---
>
> Key: HADOOP-12875
> URL: https://issues.apache.org/jira/browse/HADOOP-12875
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Hadoop-12875-001.patch, Hadoop-12875-002.patch
>
>
> This JIRA describes contract test and unit test cases support for azure data
> lake file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232755#comment-15232755 ] Chris Nauroth commented on HADOOP-12875: I just found the Azure Data Lake getting started link from the documentation in the HADOOP-12666 patch. When I get a chance, I'll try following that to see if I can get these tests running end-to-end. https://azure.microsoft.com/en-in/documentation/articles/data-lake-store-get-started-portal/ > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15231362#comment-15231362
]
Chris Nauroth commented on HADOOP-12875:
[~vishwajeet.dusane], thank you for providing a patch to make use of the
contract tests. I have a few comments in addition to the helpful feedback from
Tony.
How can I (and other Apache community members) obtain account credentials that
we can put into contract-test-options.xml, so that we can run the tests? I
have an Azure subscription. I checked manage.windowsazure.com, but I couldn't
find an option for provisioning Azure Data Lake access. It's going to be vital
for ongoing maintenance that community members have a way to get credentials so
that they can test patches against the live service before committing.
For configuration of the credentials, I recommend using a technique we've used
in hadoop-aws and hadoop-azure to split the credentials into a separate XML
file, which then gets XIncluded from the main XML file. We can then place the
name of the file with the credentials into .gitignore. This helps prevent
accidentally committing someone's private credentials to the Apache repo, which
would then compromise the account. Check out hadoop-aws and hadoop-azure for
more details on how to do this.
{code}
System.setProperty("hadoop.home.dir", System.getProperty("user.dir"));
{code}
Why is this necessary?
I'm unclear on the intent of the various "benchmark" tests. They use a mock
back-end, so they aren't really providing an accurate benchmark of the true
service interaction. There are no assertions, so they aren't verifying
functionality beyond making sure things don't throw exceptions. They print
timing information to the console, so is the expectation that these tests could
be used for manual measurement before and after applying later patches?
Your time measurements are using {{System#currentTimeMillis}}, which may be
subject to inaccuracy if the system clock changes or NTP makes a negative
adjustment in the middle of a test run. Instead, I recommend using
{{org.apache.hadoop.util.Time#monotonicNow}}, which is a wrapper over
{{System#nanoTime}}, which is guaranteed to be monotonic increasing.
{code}
@Override
protected AbstractFSContract createContract(Configuration configuration) {
try {
return new AdlStorageContract(configuration);
} catch (URISyntaxException e) {
return null;
} catch (IOException e) {
return null;
}
}
{code}
If any of these exceptions happens, then returning null is likely to cause a
confusing {{NullPointerException}} later. I'd prefer that we fail fast by
throwing an unchecked exception, such as {{IllegalStateException}}, with a
descriptive error message, and the original exception nested as root cause.
It's unusual to see contract test subclasses adding other test cases specific
to the file system, like {{readCombinationTest}}. The abstract contract test
classes are meant to fully define the test cases, and then the subclasses
usually just tweak the contract and skip tests that they aren't able to satisfy
yet. For clarity, I suggest refactoring those additional tests into separate
suites.
> [Azure Data Lake] Support for contract test and unit test cases
> ---
>
> Key: HADOOP-12875
> URL: https://issues.apache.org/jira/browse/HADOOP-12875
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Hadoop-12875-001.patch
>
>
> This JIRA describes contract test and unit test cases support for azure data
> lake file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[ https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224364#comment-15224364 ] Vishwajeet Dusane commented on HADOOP-12875: Thanks [~twu] - Good point. I will make the changes to incorporate the same. > [Azure Data Lake] Support for contract test and unit test cases > --- > > Key: HADOOP-12875 > URL: https://issues.apache.org/jira/browse/HADOOP-12875 > Project: Hadoop Common > Issue Type: Test > Components: fs, fs/azure, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > Attachments: Hadoop-12875-001.patch > > > This JIRA describes contract test and unit test cases support for azure data > lake file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15214317#comment-15214317
]
Tony Wu commented on HADOOP-12875:
--
Hi [~vishwajeet.dusane],
Thanks a lot for filing a separate JIRA, isolating out the ADL unit tests and
providing a FS contract test! This is very helpful.
I did a quick scan of the patch and have the following comments regarding the
newly added FS contract tests:
Instead of having the following change in various contract test implementations:
{code:java}
+ @Override
+ protected boolean isSupported(String feature) throws IOException {
+return true;
+ }
{code}
It's better to define a {{adls.xml}} file where you specify the file system
behavior. You can refer to {{wasb.xml}} as an example. This page
[here|https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/testing.html]
also describes the details of FS contract tests and best practices of adding
new ones.
Instead of adding {{@Ignore}} to test cases unsupported by ADL, I believe you
can use {{ContractTestUtils#unsupported("...")}} or
{{ContractTestUtils#unsupported("skip")}} instead.
> [Azure Data Lake] Support for contract test and unit test cases
> ---
>
> Key: HADOOP-12875
> URL: https://issues.apache.org/jira/browse/HADOOP-12875
> Project: Hadoop Common
> Issue Type: Test
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: Hadoop-12875-001.patch
>
>
> This JIRA describes contract test and unit test cases support for azure data
> lake file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12875) [Azure Data Lake] Support for contract test and unit test cases
[
https://issues.apache.org/jira/browse/HADOOP-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183000#comment-15183000
]
Hadoop QA commented on HADOOP-12875:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 22 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patch modules with no Java source: hadoop-tools
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 31m 38s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 32m 31s
{color} | {color:green} hadoop-tools in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791157/Hadoop-12875-001.patch
|
| JIRA Issue |
