[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128247#comment-17128247 ] Andras Bokor commented on HADOOP-15546: --- For git greppers: this was committed with the following commit message: "HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests." So grepping for HADOOP-15546 will show no result. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Thomas Marqardt >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573672#comment-16573672
]
Thomas Marquardt commented on HADOOP-15546:
---
I will commit this shortly.
+1 from me. All tests pass with fs.azure.scale.test.enabled set to true for
both WASB and ABFS.
*Tests run: 257, Failures: 0, Errors: 0, Skipped: 11*
*Tests run: 1, Failures: 0, Errors: 0, Skipped: 0*
*Tests run: 769, Failures: 0, Errors: 0, Skipped: 183*
*Tests run: 186, Failures: 0, Errors: 0, Skipped: 10*
We will address the Yetus issue later.
We will address the issues around config in a separate JIRA. For now, the
recommendation is to use separate files (azure-auth-keys.xml and
azure-bfs-auth-keys.xml) for WASB and ABFS. The contents are as shown in the
example below:
*azure-auth-keys.xml:*
fs.contract.test.fs.wasb
wasb://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net
The name of the azure file system for testing.
fs.azure.test.account.name
*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net
fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net
*{YOUR_ACCOUNT_KEY}*
fs.azure.scale.test.enabled
true
fs.azure.secure.mode
false
*azure-bfs-auth-keys.xml*:
fs.contract.test.fs.abfs
abfs://*{YOUR_FILESYSTEM_NAME}*@*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net
The name of the azure file system for testing.
fs.azure.test.account.name
*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net
fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net
*{YOUR_ACCOUNT_KEY}*
fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.dfs.core.windows.net
*{YOUR_ACCOUNT_KEY}*
fs.azure.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net
*{YOUR_ACCOUNT_KEY}*
fs.azure.test.account.key.*{YOUR_ACCOUNT_NAME}*.blob.core.windows.net
*{YOUR_ACCOUNT_KEY}*
fs.azure.scale.test.enabled
true
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Thomas Marquardt
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch,
> HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573245#comment-16573245 ] Sean Mackrory commented on HADOOP-15546: I kicked off the Yetus build again in case that Docker issue is transient - it'd be nice to see checkstyles results, etc.. It certainly doesn't appear to be related to this patch, which compiles just fine for me locally and so far tests are passing... > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573241#comment-16573241
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934762/HADOOP-15546-HADOOP-15407-012.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15004/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch,
> HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573233#comment-16573233 ] Sean Mackrory commented on HADOOP-15546: +1 to the 012 patch, I think, as it has the equals() change, a couple of others that make sense (especially in light of explanations above), and all other diffs appear to be an artifact of a different diff tool. Will retest and commit if it works as well as it did last night... > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572710#comment-16572710 ] Da Zhou commented on HADOOP-15546: -- [~mackrorysd], thanks for the review. Yes, the main purpose of the change for VersionedFileStatus is for test. HADOOP-15546-HADOOP-15407-012.patch: Added back the missing part for hadoop-common project. Retested it and all tests passed. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch, > HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572709#comment-16572709
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934762/HADOOP-15546-HADOOP-15407-012.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15002/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch,
> HADOOP-15546-HADOOP-15407-012.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572686#comment-16572686
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
11s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934758/HADOOP-15546-HADOOP-15407-011.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15001/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, HADOOP-15546-HADOOP-15407-011.patch,
> azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572601#comment-16572601
]
Sean Mackrory commented on HADOOP-15546:
Looking at the diff between patch 009 and patch 010, compilation just fails
because all the changes in hadoop-common are missing. After working around
that, I can confirm that all the tests appear to be passing now (haven't tried
the scale tests yet - they failed for me - I suspect I need to be closer to my
Azure region).
{quote}Updated equals() method in VersionedFileStatus: if versionedfilestatus
is comparing to a class inherited from FileStatus, then only path is
compared.{quote}
This set of a few red flags in my head, but I'm trying to think of when I would
compare FileStatus instances in real life, and I can't. So I suppose it's just
for testing, in which case that fulfils the purpose?
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572569#comment-16572569
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
13s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934746/HADOOP-15546-HADOOP-15407-010.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15000/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> HADOOP-15546-HADOOP-15407-010.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572146#comment-16572146 ] Thomas Marquardt commented on HADOOP-15546: --- Da and I are reviewing the latest patch today. We'll make changes to fix any failing tests, and hope to commit the change later today. We'll also address all the comments above. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571091#comment-16571091 ] Sean Mackrory commented on HADOOP-15546: As a data point, prior to applying this patch, the ListStatus / FileStatus tests work, but the BackCompat test still fails, and the other failures mentioned above are either skipped or not being run. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, > azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571012#comment-16571012
]
Steve Loughran commented on HADOOP-15546:
-
[~tmarquardt] the only key conflict that is really surfacing is
{{fs.azure.test.account.name}}. How about a new key is used there; the old one
used solely for the Wasb connector suites.
* {{fs.azure.test.account.name}} set: run wasb tests
* {{fs.abfs.test.account.name}} set: run abfs tests
* both set: run all tests.
makes it easy to do full test runs while doing abfs dev by leaving the azure
key unset; regression testing of changes in hadoop-common, poms, shared tests
can be tested by running both.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch,
> azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571010#comment-16571010
]
Steve Loughran commented on HADOOP-15546:
-
HADOOP-15546 patch 009
* resource azure-bfs-test.xml imports azure-bfs-auth-keys.xml so you can set
fs.azure.test.account.name to an abfs account
* and the abfs tests check that this is true
* fix test which didn't compile with the touch() call in the superclass
* log4j.properties back to logging all at debug. noisier, but sets you up to
debug things.
* ITestAzureNativeContractAppend.testAppendDirectory() expects the exception
raised by that FS
The sole reason for needing a separate {{azure-bfs-auth-keys.xml}} seems to be
for reusing the {{fs.azure.test.account.name}}
key to refer to an abfs account with the abfs tests. Given this doesn't get
along with the waze tests, I don't understand why
this is needed. Can't we just have a separate config key , say
{{fs.abfs.test.account.name}} for the abfs tests? That will
also provide an easy way to skip all the wasb tests in the module: simply
undefine {{fs.azure.test.account.name}} from the config resources
Testing: wasb against UK; abfs against US. I'm in the US right now and a lot of
the WASB tests were timing out in a (very slow) parallel test run,
The only real outlier here was {{ITestBlobDataValidation}}, where the assertion
that MD5 checksums were coming back failed. This happens
repeatedly, even when run as a single test. I have no idea what is going on
here.
{code}
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 21.424
s <<< FAILURE! - in org.apache.hadoop.fs.azure.ITestBlobDataValidation
[ERROR] testCheckBlockMd5(org.apache.hadoop.fs.azure.ITestBlobDataValidation)
Time elapsed: 2.541 s <<< ERROR!
java.io.IOException: java.lang.AssertionError Please see the cause for further
information.
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.close(NativeAzureFileSystem.java:1058)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:235)
at
org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Suppressed: java.io.IOException: java.lang.AssertionError Please see
the cause for further information.
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.close(BlobOutputStreamInternal.java:317)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
... 17 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.AssertionError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:530)
... 19 more
Caused by: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570685#comment-16570685 ] Thomas Marquardt commented on HADOOP-15546: --- I'm back from paternity leave. Sorry for the long pause. The preview of ABFS shared many of the same config property names as WASB, and we now have a lot of customers using them. I agree that we need to improve the configuration, make it easier to use, and easier to troubleshoot (better error handling). However, we cannot remove or rename the current config properties without breaking our customers. I spoke with Steve about this on a phone call this morning, and would like you to make a list of the changes you'd like to see regarding configuration and we will work on them in a separate JIRA. For now, I'd like to revert back to using two test configuration files, one for WASB and one for ABFS. I promise we will improve the situation, but I want to do it right, and need sometime to think thru it. I don't want to block progress in the meantime. So our primary goal is to stabilize ABFS (get all the tests passing, bugs fixed, etc), and then we can fix the config issue. ABFS was originally designed to replace WASB, which is why it uses the same config. The idea was that users would simply replace the class mapping for the WASB scheme, and they'd suddenly be using the new ABFS code. We have since realized that ABFS needs to be independent of WASB, allow WASB to be an option, even though it will be deprecated and all future development work will target ABFS. So we will think about this and come up with a way to onboard customers to a better config solution for ABFS, but for now please do not remove/rename or change the behavior of existing config keys, as this will break customers. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570664#comment-16570664
]
Sean Mackrory commented on HADOOP-15546:
So with this small change I was able to get WASB and ABFS tests working. We
could leave the other properties as fs.azure.test because they should usually
be the same between WASB and ABFS. This is the only case I see where the values
MUST be different for the 2, and we don't have any way to specify 2 different
values at once unless we rename that property, so azure -> azure-bfs to resolve
the conflict. I don't think it's very clean, but the alternatives all seem
messier.
{code}
diff --git
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/constants/TestConfigurationKeys.java
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/constants/TestConfigurationKeys.java
index a2251ee..704b0b2 100644
---
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/constants/TestConfigurationKeys.java
+++
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/constants/TestConfigurationKeys.java
@@ -26,7 +26,7 @@
*/
@InterfaceStability.Evolving
public final class TestConfigurationKeys {
- public static final String FS_AZURE_TEST_ACCOUNT_NAME =
"fs.azure.test.account.name";
+ public static final String FS_AZURE_TEST_ACCOUNT_NAME =
"fs.azure-bfs.test.account.name";
public static final String FS_AZURE_TEST_ACCOUNT_KEY_PREFIX =
"fs.azure.test.account.key.";
public static final String FS_AZURE_TEST_HOST_NAME =
"fs.azure.test.host.name";
public static final String FS_AZURE_TEST_HOST_PORT =
"fs.azure.test.host.port";
{code}
This leaves me with a much cleaner list of test failures. Any thoughts on
including this in the current patch or resolving this conflict some other way?
* 3/5 tests fail in ITestWasbAbfsCompatibility and they look legit: 1 file
found when expecting 2, and 404's on some file paths.
* ITestAzureBlobFileSystemBackCompat also fails with a 404.
* ITestAzureBlobFileSystemRenameUnicode.testRenameFileUsingUnicode fails (gets
an empty listing)
* ITestAzureBlobFileSystemFileStatus.testEnsureStatusWorksForRoot and
ITestAzureBlobFileSystemListStatus.testListFiles both fail because of otherwise
identical instances of VersionedFileStatus and LocatedFileStatus not being
equal (only the type differs, and the version is null).
* 16/20 tests failed in ITestFileSystemOperationsWithThreads, with "No ...
operation found in logs"
I suspect the first 3 items are probably something server-side, so I'm going to
start looking into the bottom 2.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560511#comment-16560511 ] Sean Mackrory commented on HADOOP-15546: So [~ste...@apache.org] - I like the idea of combining config files if we're going to have ABFS and WASB coexist in the same module, but I'd like to be able to fill it out such that I can run all the tests in that module. What about having a separate properties fs.wasb.test.account.name and fs.abfs.test.account.name or something? > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553395#comment-16553395 ] Da Zhou commented on HADOOP-15546: -- Hi [~mackrorysd], although both WASB and ABFS have property "fs.azure.test.account.name", their values are different. For WASB, its value is: ACCOUNT_NAME.*blob*.core.windows.net For ABFS, its valueis : ACCOUNT_NAME.*dfs*.core.windows.net So when merging WASB and ABFS configuration file and making them share same properties (eg: fs.azure.test.account.name) , their configuration parser also need to be updated to make sure they are using correct full account name. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553285#comment-16553285 ] Sean Mackrory commented on HADOOP-15546: At least some of the WASB issues are coming from the overloading of the fs.azure.test.account.name property. If I set that to .blob.core.windows.net, the WASB unit tests pass. But then more of the ABFS integration tests start failing earlier. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553256#comment-16553256 ] Sean Mackrory commented on HADOOP-15546: Upon looking at this with fresh eyes, I see the above failures were all in the original WASB package (org.apache.hadoop.fs.azure). There are other failures within the new org.apache.hadoop.fs.azurebfs package, but they are distinct. I posted the stack traces here: https://gist.github.com/mackrorysd/dc0abebfea74f394e184e392215b47fe. Looks like the contract tests are mostly running and passing. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553129#comment-16553129 ] Da Zhou commented on HADOOP-15546: -- I'm looking into this, will update it once I find the cause. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552151#comment-16552151 ] Shane Mainali commented on HADOOP-15546: [~DanielZhou], can you help take a look to see if there's something wrong with the configuration? [~tmarquardt] was looking optimistically during his OOF but we should look further. From discussion with [~tmarquardt], there are a few things to address probably but it shouldn't be a major effort. > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550941#comment-16550941
]
Sean Mackrory commented on HADOOP-15546:
Had to change uses of touch(FileSystem, Path) to touch(Path), fyi. Other
changes look good. Hoping to dig into this today, but with my config that's
virtually identical to what mostly works without this patch, every single ABFS
integration test fails for me with one of the following after applying this
patch (and I have moved it from azure-bfs-auth-keys.xml to azure-auth-keys.xml):
{code}
Caused by: com.microsoft.azure.storage.StorageException: Value for one of the
query parameters specified in the request URI is invalid.
Caused by: com.microsoft.azure.storage.StorageException: Server failed to
authenticate the request. Make sure the value of Authorization header is formed
correctly including the signature.
{code}
It's mostly the former. Attaching a sanitized copy of my config in case you
happen to spot something that's obviously wrong or that has changed before I do.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550088#comment-16550088
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 8m
7s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932346/HADOOP-15546-HADOOP-15407-008.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14917/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch,
> HADOOP-15546-HADOOP-15407-008.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550079#comment-16550079 ] Steve Loughran commented on HADOOP-15546: - HADOOP-15546 patch 008 * review ITestAzureBlobFileSystemListStatus * move unicode test in ITestAzureBlobFileSystemRename into parameterized test suite ITestAzureBlobFileSystemRenameUnicode; isolates failures better. (left the old one there for now) * ITestReadWriteAndSeek -> ITestAbfsReadWriteAndSeek * helper methods touch(path), write(path, data) in AbstractAbfsIntegrationTest; use where appropriate * all tests of > a megabyte or two tagged as scale, so splitting up "basic fast set" from "the full diligence set". People must still due that scale test before a final commit, but it means that basic dev cycles are shorter Ignoring the parallel test issue (which I am), I'm pretty much done here. If things are failing, I don't think it's because this patch breaks stuff, it just surfaces some problems. Test results * ITestAzureBlobFileSystemRenameUnicode fails * All the wasb tests are failing with an NPE inside wasb using the login details which I am using to run the abfs tests. (as shown above) * new test which compare LocatedFileStatus to VersionedFileStatus fail if it VersionedFileStatus is on the left of the .equals(), as it requires a matching type and version. To date, all FileStatus subclasses just compare on path. *I don't know if this is going to be a problem*. I think this is ready to go in. Yes, stuff fails, but this gets the tests runnable by others, so we can make progress on the branch. Reviews please! > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, > HADOOP-15546-HADOOP-15407-008.patch > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549868#comment-16549868
]
Steve Loughran commented on HADOOP-15546:
-
# rename() is working today. Assume: networking issues somewhere
# getting failure in existing wasb test
{{TestNativeAzureFileSystemBlockCompaction}}
{code}
[ERROR]
testCompactionDisabled(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction)
Time elapsed: 0.547 s <<< ERROR!org.apache.hadoop.fs.azure.AzureException:
com.microsoft.azure.storage.StorageException: Value for one of the query
parameters specified in the request URI is invalid.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2141)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getAncestor(NativeAzureFileSystem.java:3002)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1786)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1634)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
at
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction.testCompactionDisabled(TestNativeAzureFileSystemBlockCompaction.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: com.microsoft.azure.storage.StorageException: Value for one of the
query parameters specified in the request URI is invalid.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:209)
at
com.microsoft.azure.storage.blob.CloudBlobContainer.downloadAttributes(CloudBlobContainer.java:570)
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobContainerWrapperImpl.downloadAttributes(StorageInterfaceImpl.java:255)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.checkContainer(AzureNativeFileSystemStore.java:1268)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2057)
... 20 more
Caused by: java.lang.NullPointerException
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:189)
{code}
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2.
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548620#comment-16548620
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
10s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932163/HADOOP-15546-HADOOP-15407-007.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14909/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548555#comment-16548555
]
Steve Loughran commented on HADOOP-15546:
-
code
{code}
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.933
s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractMkdir
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameDirIntoExistingDir:150->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameDirIntoExistingDir:154->AbstractFSContractTestBase.assertIsFile:316
» FileNotFound
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileNonexistentDir:171->AbstractFSContractTestBase.rename:363
»
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameFileNonexistentDir:171->AbstractFSContractTestBase.rename:363
»
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameNewFileSameDir:45->AbstractFSContractTestBase.rename:363
»
[ERROR]
ITestAbfsFileSystemContractRename>AbstractContractRenameTest.testRenameNewFileSameDir:45->AbstractFSContractTestBase.rename:363
»
[INFO]
{code}
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548549#comment-16548549
]
Steve Loughran commented on HADOOP-15546:
-
HADOOP-15446 patch 007
more test code review;
* moving stream reads/writes into try-with-resources
* assertEquals order
* messages on assert failures
* clean up unused imports
* ITestAzureBlobFileSystemRandomRead tagged as scale test
* Address Sean Macrory's comment about errors in testing doca
I am seeing {{ITestAbfsFileSystemContractRename}} fail repeatedly with timeouts
and errors. That is: rename() doesn't appear to be working.
would be good to know what others see
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548374#comment-16548374 ] Steve Loughran commented on HADOOP-15546: - thx for your comments; will look at them. I want to get rid of the second file. I'm actually xincluding both to my real auth-keys file which lives outside the source tree entirely > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, > HADOOP-15546-HADOOP-15407-006.patch > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548291#comment-16548291
]
Sean Mackrory commented on HADOOP-15546:
The patch looks pretty good to me from a code standpoint, and fixes a couple of
things I initially added to my list of to-dos (like some of the config file
inconsistency and imports). One nit: on line 2619 of the latest patch, the
config key is missing the second 's' in 'abfss':
{code}
+
+ fs.contract.test.fs.abfs
+ abfss://testcontai...@accountname.dfs.core.windows.net
+ Container for contract tests
+
{code}
I'm having some serious issues running with this patch that I need to track
down. I thought I had everything set up correctly because without this patch I
can run the tests and I have 2-3 failures, depending on the run. With this
patch, most of them fail because the URL they're trying doesn't exist. Trying
to track down why. I've symlinked between azure-auth-keys.xml and
azure-bfs-auth-keys.xml, and manually messed with that, so I don't *think* it's
just not finding my configs, but I'll need to dig more.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548133#comment-16548133
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931670/HADOOP-15546-HADOOP-15407-006.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14902/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548092#comment-16548092
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931670/HADOOP-15546-HADOOP-15407-006.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14901/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547355#comment-16547355
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931670/HADOOP-15546-HADOOP-15407-006.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14899/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544326#comment-16544326
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 54 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15407 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
41s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 38m
35s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m
13s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} HADOOP-15407 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 39m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 39m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
46s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
52s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931666/HADOOP-15546-HADOOP-15407-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 2422ffe4eb54 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544316#comment-16544316
]
Steve Loughran commented on HADOOP-15546:
-
HADOOP-15446 patch 006: what does append(dir) do?
pulling up the abfs {{testAppendDirectory}} test to append onto a dir has shown
up that the correct exception wasn't stated: HDFS expects
FileAlreadyExistsException.
* fix that in filesystem.md
* fix localFs to throw that exception, rather than IOE, without typo in message
* contract test to catch IOE but expect FileAlreadyExistsException as the
"strict" outcome; handle others as "relaxed"
I haven't changed the abfs append (it raises FileNotFoundException) as that's
valid, just different from HDFS.
(Yes, this is mixing in some changes in contract-tests w/ the ABFS tests but
these new tests from Microsoft are showing up new corner cases which need
better coverage in docs & contract tests). We could split out into a
self-contained patch once happy
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch,
> HADOOP-15546-HADOOP-15407-006.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent
> to contract test setup, and general work to make the tests work well over
> slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544315#comment-16544315 ] Steve Loughran commented on HADOOP-15546: - oh, that stack trace from the test also has a typo " Cannot append to a diretory" will fix > ABFS: tune imports & javadocs; stabilise tests > -- > > Key: HADOOP-15546 > URL: https://issues.apache.org/jira/browse/HADOOP-15546 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: HADOOP-15407 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15546-001.patch, > HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, > HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, > HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch > > > Followup on HADOOP-15540 with some initial review tuning > h2. Tuning > * ordering of imports > * rely on azure-auth-keys.xml to store credentials (change imports, > docs,.gitignore) > * log4j -> info > * add a "." to the first sentence of all the javadocs I noticed. > * remove @Public annotations except for some constants (which includes some > commitment to maintain them). > * move the AbstractFS declarations out of the src/test/resources XML file > into core-default.xml for all to use > * other IDE-suggested tweaks > h2. Testing > Review the tests, move to ContractTestUtil assertions, make more consistent > to contract test setup, and general work to make the tests work well over > slower links, document, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544268#comment-16544268
]
Steve Loughran commented on HADOOP-15546:
-
Test failure was about from the append test I'd pulled up into common; either
test is wrong or failure not quite as expected
{code}
[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.735 s
<<< FAILURE! - in
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
[ERROR]
testAppendDirectory(org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractAppend)
Time elapsed: 0.017 s <<< ERROR!
java.io.IOException: Cannot append to a diretory
(=file:/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/3/ftiIMjcjYy/test/target
)
at
org.apache.hadoop.fs.RawLocalFileSystem.append(RawLocalFileSystem.java:281)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1430)
at
org.apache.hadoop.fs.contract.AbstractContractAppendTest.lambda$testAppendDirectory$3(AbstractContractAppendTest.java:164)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:491)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:377)
at
org.apache.hadoop.fs.contract.AbstractContractAppendTest.testAppendDirectory(AbstractContractAppendTest.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch,
> HADOOP-15546-HADOOP-15407-005.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> No actual code changes here; just setting things up better for >1 person
> editing & testing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543880#comment-16543880
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 54 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15407 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
55s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
30s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m
25s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} HADOOP-15407 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 31m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
25s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 19s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.contract.rawlocal.TestRawlocalContractAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931609/HADOOP-15546-HADOOP-15407-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux e266e37598af 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541958#comment-16541958
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15407 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
20s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 39m
7s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
47s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} HADOOP-15407 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
25s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
57s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931335/HADOOP-15546-HADOOP-15407-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux 83bd65058dd6 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-15407 / fb17346 |
| maven | version: Apache Maven 3.3.9 |
| Default Java |
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533090#comment-16533090
]
genericqa commented on HADOOP-15546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15407 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
36s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 26m
18s{color} | {color:red} root in HADOOP-15407 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
35s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} HADOOP-15407 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 22m
13s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 22m 13s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
19s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930316/HADOOP-15546-HADOOP-15407-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux 0c31285d58dd 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-15407 / 538fcf8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java |
[jira] [Commented] (HADOOP-15546) ABFS: tune imports & javadocs; stabilise tests
[
https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533057#comment-16533057
]
Steve Loughran commented on HADOOP-15546:
-
HADOOP-15546 patch 003
* assume messages are meaningful when config is incorrect/incomplete
* URI errors reported meaningfully. Changed text of InvalidUriException so it
parses well from all constructed copies.
* ABFS toString() useful in logs and assertions
* VersionedFileStatus.toString() to include version as well as all FileStatus
fields.
ITestAzureBlobFileSystemFileStatus fixed to
* pass when owner != group (ie. stevel/staff)
* split file and root validation into two methods
* validate root group too
I've then gone and broken
{{ITestAzureBlobFileSystemFileStatus.testEnsureStatusWorksForRoot()}} by
verifying that {fs.listStatus("/")[0].equals(fs.getFileStatus("/"))}}.
Normally FileStatus.equals only compares paths, not here, and the version
fields don't match.. I do not know the larger-scale implications of this
inequality.
Tested: Azure ireland. Got {{ITestAzureBlobFileSystemFileStatus}} to work
excluding the new assertion added.
> ABFS: tune imports & javadocs; stabilise tests
> --
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: HADOOP-15407
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15546-001.patch,
> HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch,
> HADOOP-15546-HADOOP-15407-003.patch
>
>
> Followup on HADOOP-15540 with some initial review tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports,
> docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some
> commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file
> into core-default.xml for all to use
> * other IDE-suggested tweaks
> No actual code changes here; just setting things up better for >1 person
> editing & testing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
