[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780336#comment-16780336 ] Masatake Iwasaki commented on HADOOP-16068: --- Thans for the update, [~ste...@apache.org]. +1. tests passed on Japan region. Added documentation would be quite helpful for users and contributors. > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch, > HADOOP-16068-009.patch, HADOOP-16068-010.patch, HADOOP-16068-011.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779364#comment-16779364
]
Steve Loughran commented on HADOOP-16068:
-
checkstyle
{code}
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/TestCustomOauthTokenProvider.java:32:import
static org.apache.hadoop.fs.azurebfs.extensions.WrappingTokenProvider.*;:
Using the '.*' form of import should be avoided -
org.apache.hadoop.fs.azurebfs.extensions.WrappingTokenProvider.*.
[AvoidStarImport]
{code}
I'm not worried about that for static imports, but will switch it just to keep
the complaints down. Other than that, this patch is ready to go in unless
someone has fundamental issues with the proposed interface...and if they do, we
can change that after. The key thing is that this improves the binding of auth
and DT issuing by allowing them to use the store URI to bind
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch,
> HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch,
> HADOOP-16068-009.patch, HADOOP-16068-010.patch, HADOOP-16068-011.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778128#comment-16778128
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 9 unchanged - 0 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 101 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960197/HADOOP-16068-011.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 31d66b4c3224 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 585cebf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15977/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15977/artifact/out/whitespace-eol.txt
|
| Test Results |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777859#comment-16777859 ] Steve Loughran commented on HADOOP-16068: - thanks for the comments, will look at. Nice see the tests worked. That TestCustomOauthTokenProvider test is designed to be a unit test; it should be running on jenkins. But I look at jenkins & see it is skipping; need to understand what I can do to make sure it will run, always > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch, > HADOOP-16068-009.patch, HADOOP-16068-010.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777086#comment-16777086
]
Masatake Iwasaki commented on HADOOP-16068:
---
LGTM overall. Added tests including ITestAbfsDelegationTokens succeeded in
Japan region. Some nits found on patch walk through.
AbfsDtFetcher.java: javadoc of getServiceName (just copied from HdfsDtFetcher)
should be updated.
src/main/resources/META-INF/services/org.apache.hadoop.security.token.DtFetcher:
having duplicate entries.
TestCustomOauthTokenProvider.java: subclass of AbstractAbfsIntegrationTest
should be prefixed wth ITest*?
abfs.md:
{code:java}
The [az
storage](https://docs.microsoft.com/en-us/cli/azure/storage/account?view=azure-cli-latest)
subcommand
handles all storage commands, [`az storage account
create`](https://docs.microsoft.com/en-us/cli/azure/storage/account?view=azure-cli-latest#az-storage-account-create)
{code}
Intended target of the first link would be
[https://docs.microsoft.com/en-us/cli/azure/storage?view=azure-cli-latest].
{code:java}
You can list locations from az account list-locations
{code}
Following cmdlines does not show usage of list-locations.
{code:java}
or you can configure an identity to be used only for a specific storage account
with
`fs.azure.account.oauth2.client.id.\.dfs.core.windows.net`.
{code}
You don't need backslash here.
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch,
> HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch,
> HADOOP-16068-009.patch, HADOOP-16068-010.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777039#comment-16777039
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 100 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960040/HADOOP-16068-010.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux a020e315b8f3 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6cec906 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15969/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15969/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776930#comment-16776930
]
Steve Loughran commented on HADOOP-16068:
-
log on oauth returns HTML for next patch. Doesn't fix problem, but helps
{code}
bin/hadoop fs -ls abfs://contai...@abfswales1.dfs.core.windows.net/
14:22:01
2019-02-25 14:23:07,975 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:initialize(101)) - Initializing AzureBlobFileSystem
for abfs://contai...@abfswales1.dfs.core.windows.net/
2019-02-25 14:23:08,182 [main] DEBUG services.AbfsClientThrottlingIntercept
(AbfsClientThrottlingIntercept.java:initializeSingleton(62)) - Client-side
throttling is enabled for the ABFS file system.
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:getFileStatus(434)) -
AzureBlobFileSystem.getFileStatus path:
abfs://contai...@abfswales1.dfs.core.windows.net/
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:performAbfsAuthCheck(1101)) - ABFS authorizer is not
initialized. No authorization check will be performed.
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystemStore
(AzureBlobFileSystemStore.java:getIsNamespaceEnabled(200)) - Get root ACL status
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(AccessTokenProvider.java:isTokenAboutToExpire(77)) - AADToken: no token.
Returning expiring=true
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(AccessTokenProvider.java:getToken(48)) - AAD Token is missing or expired:
Calling refresh-token from abstract base class
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(ClientCredsTokenProvider.java:refreshToken(57)) - AADToken: refreshing
client-credential based token
2019-02-25 14:23:08,274 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenUsingClientCreds(94)) - AADToken: starting
to fetch token using client creds for client ID
40c8b4e5-865a-4297-bbbd-195df2a8a806
2019-02-25 14:23:08,274 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenSingleCall(297)) - Requesting an OAuth token
by POST to
https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize
2019-02-25 14:23:08,275 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Request Headers
2019-02-25 14:23:08,275 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Connection=close
2019-02-25 14:23:08,872 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenSingleCall(319)) - Response 200
2019-02-25 14:23:08,872 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Response Headers
2019-02-25 14:23:08,873 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - HTTP Response=HTTP/1.1 200 OK
2019-02-25 14:23:08,873 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Server=Microsoft-IIS/10.0
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - X-Content-Type-Options=nosniff
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Connection=close
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Pragma=no-cache
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - P3P=CP="DSP CUR OTPi IND OTRi
ONL FIN"
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Date=Mon, 25 Feb 2019 14:23:08
GMT
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - X-Frame-Options=DENY
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
Strict-Transport-Security=max-age=31536000; includeSubDomains
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Cache-Control=no-cache,
no-store
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
Set-Cookie=stsservicecookie=ests; path=/; secure;
HttpOnly;x-ms-gateway-slice=prod; path=/; secure;
HttpOnly;esctx=AQABAACEfexXxjamQb3OeGQ4GugvJnqCleb5fTHXUNtG2e2vTfvF4cV_3pPI9WICNEPT-85W2F8bAZgHI_L1btZuYPY5pru5civf8jCRgzmUuU5mfcyeeYApPXDdetus1cHPtPEe7cNjOewm_S0dVIvGvA3pS87ggExUYAXwnRzRj55Z1T7ftN444Vg3QHApA6jYvewgAA;
domain=.login.microsoftonline.com; path=/; secure;
HttpOnly;fpc=AqAbRMgiP0pMldA4zkrmUY36f-C2AQAAAEzxBdQO; expires=Wed,
27-Mar-2019 14:23:08 GMT; path=/; secure;
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776433#comment-16776433 ] Steve Loughran commented on HADOOP-16068: - Also happens if your password expires. The keys you get with an a/c have an expiry (which makes sense), but the error message doesn't mention that. Will add it to the docs. > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch, > HADOOP-16068-009.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776140#comment-16776140
]
Masatake Iwasaki commented on HADOOP-16068:
---
{code:java}
2019-02-14 14:32:09,549 [main] INFO util.ExitUtil
(ExitUtil.java:terminate(210)) - Exiting with status -1: Operation failed:
"Server failed to authenticate the request. Make sure the value of
Authorization header is formed correctly including the signature.", 403, HEAD,
https://account.dfs.core.windows.net/stevel-testing2?resource=filesystem=90
{code}
I can reproduce the error by running azure tests on my VM with skewed clock.
Synced clock fixed this.
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch,
> HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch,
> HADOOP-16068-009.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775552#comment-16775552
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 92 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959839/HADOOP-16068-009.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux acdb0d48f896 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed13cf8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15965/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15965/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775515#comment-16775515 ] Steve Loughran commented on HADOOP-16068: - Patch 009 * docs now cover account setup, container creation, some more errors * contains fix for HADOOP-16139 > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch, > HADOOP-16068-009.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772308#comment-16772308
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 77 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959316/HADOOP-16068-008.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 0c71fbe7ca7c 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 02d04bd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15940/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15940/testReport/ |
| Max. process+thread count | 295 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772262#comment-16772262 ] Steve Loughran commented on HADOOP-16068: - +rendered on github: https://github.com/steveloughran/hadoop/blob/abfs/HADOOP-16068-Delegation/hadoop-tools/hadoop-azure/src/site/markdown/abfs.md > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768586#comment-16768586
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 75 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958745/HADOOP-16068-007.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 79f2a3390a29 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0d7a5ac |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15923/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15923/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768442#comment-16768442
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 64 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958724/HADOOP-16068-006.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 5e3049fc6770 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7a57974 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15922/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15922/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768366#comment-16768366
]
Steve Loughran commented on HADOOP-16068:
-
Tested: azure amsterdam. All hadoop-azure unit tests worked.
I am seeing failures on the command line, but I'm not sure that this problem
hasn't existed before, it's just I've not used the abfs:// codepath on "hadoop
fs"
{code}
2019-02-14 14:32:08,991 [main] INFO diag.StoreDiag
(DurationInfo.java:(53)) - Starting: Creating filesystem
abfs://stevel-testi...@account.dfs.core.windows.net/
2019-02-14 14:32:09,075 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:initialize(101)) - Initializing AzureBlobFileSystem
for abfs://stevel-testi...@account.dfs.core.windows.net/
2019-02-14 14:32:09,300 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:fileSystemExists(856)) -
AzureBlobFileSystem.fileSystemExists uri:
abfs://stevel-testi...@account.dfs.core.windows.net
2019-02-14 14:32:09,301 [main] DEBUG azurebfs.AzureBlobFileSystemStore
(AzureBlobFileSystemStore.java:getFilesystemProperties(254)) -
getFilesystemProperties for filesystem: stevel-testing2
2019-02-14 14:32:09,540 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) - =HTTP/1.1 403 Server failed to
authenticate the request. Make sure the value of Authorization header is formed
correctly including the signature.
2019-02-14 14:32:09,540 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) - x-ms-version=2018-11-09
2019-02-14 14:32:09,540 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) - Server=Windows-Azure-HDFS/1.0
Microsoft-HTTPAPI/2.0
2019-02-14 14:32:09,541 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) -
x-ms-error-code=AuthenticationFailed
2019-02-14 14:32:09,541 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) -
x-ms-request-id=c1260275-e01f-009e-7272-c4fef800
2019-02-14 14:32:09,541 [main] DEBUG services.AbfsHttpOperation
(AbfsHttpOperation.java:processResponse(277)) - Date=Thu, 14 Feb 2019 14:32:09
GMT
2019-02-14 14:32:09,541 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(198)) - HttpRequest:
403,,cid=a63bd35f-9afd-44ae-be41-268b16045013,rid=c1260275-e01f-009e-7272-c4fef800,connMs=23,sendMs=0,recvMs=208,sent=0,recv=0,HEAD,https://account.dfs.core.windows.net/stevel-testing2?resource=filesystem=90
2019-02-14 14:32:09,547 [main] INFO diag.StoreDiag
(DurationInfo.java:close(100)) - Creating filesystem
abfs://stevel-testi...@account.dfs.core.windows.net/: duration 0:00:558
Operation failed: "Server failed to authenticate the request. Make sure the
value of Authorization header is formed correctly including the signature.",
403, HEAD,
https://account.dfs.core.windows.net/stevel-testing2?resource=filesystem=90
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:135)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getFilesystemProperties(AbfsClient.java:209)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFilesystemProperties(AzureBlobFileSystemStore.java:259)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.fileSystemExists(AzureBlobFileSystem.java:859)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3324)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3373)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3341)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:832)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:420)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1127)
at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1136)
at storediag.main(storediag.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768281#comment-16768281 ] Steve Loughran commented on HADOOP-16068: - HADOOP-16068 patch 006 * checkstyle of patch 005 * pull bind() method of Stub DT manager above static methods. * change how TokenAccessProviderException inits init cause, so no need to change signature of AzureBlobkFileSystemException (==100% linking consistency across library versions) * AbfsHttpOperation lists headers on DEBUG log level only. Guess who has been having "Server failed to authenticate the request" responses. I've also started to write the docs which are so sadly lacking on this connector (what are namespaces? Why so many auth options? and how to configure them?). That'll be coming separately > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch, > HADOOP-16068-006.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766639#comment-16766639
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 5
new + 9 unchanged - 0 fixed = 14 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 63 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958471/HADOOP-16068-005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 635c261cd804 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3dc2523 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15918/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15918/artifact/out/whitespace-eol.txt
|
| Test Results |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766604#comment-16766604 ] Steve Loughran commented on HADOOP-16068: - [~DanielZhou]: thanks, fixed that Patch 005 * An ExtensionHelper class removes the repetition of probes for type, casting and invocation. Note use of Optional & how Java's checked exceptions cripples its full exploitation. * Tests for the custom OAuth provider chain * tests to verify the unbound DT provider works. * fix javadocs & style from patch 005 * TokenAccessProviderException passes cause parameter to superclass * Custom/mock Oauth token provider with unit test of construction & init. For that TokenAccessProviderException to stop losing its cause I had to add a new constructor to AzureBlobFileSystemException of type (String, Throwable). I've kept the existing (String, Exception), so that anything compiled against the old binaries will still link. I could change how TokenAccessProviderException does the cause setting (via initCause) if you'd prefer -it won't change the signature of either exception. Tested: Azure amsterdam. I've just changed the javadocs & style after doing that, so I may have just broken something. > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765324#comment-16765324 ] Da Zhou commented on HADOOP-16068: -- Thanks for adding this, great documentation and tests! There is one typo in AbfsDtFetcher.java: line 34: S3A should be replaced with ABFS. > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765041#comment-16765041
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 9
new + 7 unchanged - 0 fixed = 16 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} hadoop-tools_hadoop-azure generated 5 new + 0
unchanged - 0 fixed = 5 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12958249/HADOOP-16068-004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 38d1a1c42e30 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e7d1ae5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15908/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764982#comment-16764982 ] Steve Loughran commented on HADOOP-16068: - HADOOP-16068 patch 004 * custom oath provider can declare a UA field which is then used in the ABFS client UA header (for audit logs when the login credentials have a more complex provenance) * fix for HADOOP-16103 (more UGI resets) * testing: Azure Amsterdam; found HADOOP-16103. more resetUGIs *seems* to fix this * not done: tests to verify that the classic lifecycle works. The stub class for testing is there. I'm going to do another iteration here and pause. I think the code is OK, but it is hitting the limits of testability —primarily because the extension points for DT issuing rely on kerberos being running, which confuses other tests. # need some DT lifecycle tests which verify that the "classic" lifecycle still works; # To test the login extension point, without doing a full HADOOP-14556-style token marshalling plugin, this will need a proxy to the standard OAuth login. > ABFS Auth and DT plugins to be bound to specific URI of the FS > -- > > Key: HADOOP-16068 > URL: https://issues.apache.org/jira/browse/HADOOP-16068 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch, > HADOOP-16068-003.patch, HADOOP-16068-004.patch > > > followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind > the plugins to the specific FS instance. Without that you can't have per FS > auth > +add a stub DT plugin for testing, verify that DTs are collected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764955#comment-16764955
]
Steve Loughran commented on HADOOP-16068:
-
This triggers a failure elsewhere in the code, on account of the UGI thinking
the user is still logged in
{code}
[ERROR]
testNoOpWhenSettingSuperUserAsdentity(org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer)
Time elapsed: 0 s <<< ERROR!
java.io.IOException: There is no primary group for UGI
alice/localh...@example.com (auth:KERBEROS)
at
org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1601)
at
org.apache.hadoop.fs.azurebfs.ITestAbfsIdentityTransformer.(ITestAbfsIdentityTransformer.java:67)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:217)
at
org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:266)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763096#comment-16763096
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
26s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 8
new + 6 unchanged - 0 fixed = 14 total (was 6) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 46 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
19s{color} | {color:red} hadoop-tools_hadoop-azure generated 8 new + 0
unchanged - 0 fixed = 8 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12957959/HADOOP-16068-003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 678cf8fc6f8a 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 75e8441 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15893/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt
|
| compile |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763049#comment-16763049
]
Steve Loughran commented on HADOOP-16068:
-
HADOOP-16068 patch 003
* when DTs are on, custom DT is asked for canonical service name.
* Classic binding mechanism (used somewhere in HD/I) works with canonical
service name set by FileSystem default mechanism (uses IPAddr of azure a/c)
* if a token returned already sets a Kind, don't overwrite it with a local
instance (allows for independent unmarshalling, renewal, cancel)
* test of the binding process using a stub token manager
* Integration tests verify issue for jobs, save & load, no regression of
current code
* adds dtutils support for abfs and abfss URLs
* Addresses checkstyle issues of patch 002
The big piece of work is those integration tests; based on the S3A DT test
suites & verifying that DTs are created during job submission and the hadoop
dtutils command. The {{delegationTokenManager}} instance of the filesystem has
to be made accessible to the tests, which is done via a package-private
accessor; a method in AbstractAbfsIntegrationTest then provides that value to
any subclass which wants it, irrespective of which package they are in.
oh, and log4j for test runs turns off ShellBasedUnixGroupsMapping logging at
warning, as it keeps complaining that the newly created user,
"al...@example.com" doesn't have a user mapping.
Testing: ABFS Ireland. I'm getting complaints from some tests about not running
with namespaces, but the new test works; both in IDE-runs and in parallel test
runs (i.e both JUnit JVM lifecycles)
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762878#comment-16762878
]
Steve Loughran commented on HADOOP-16068:
-
one aspect of this patch is that it returns the FS URI in a call to
AzureBlobFileSystem.getCanonicalServiceName() when DTs are turned on.
I'd always believed that you wouldn't get anything here, but actually the base
class seems to return "52.239.143.194:0", which is the IPAddr of the store;
with the new code the FS service name would become something like
abfs://abfs-testcontainer-57aa1ca6-761b-4798-92ca-d05a9105f...@abfsaccount.dfs.core.windows.net
i..e unique for each FS instance.
It is going to mark a change in behaviour. We'll need to talk to the
implementors of any existing use of the DT plugin to see what
Today the canonical service name of *all* ABFS filesystems will be that of the
IPAddr of the {{abfsaccount.dfs.core.windows.net}} value, which turns out to be
(for me). something in amsterdam:
{code}
dfs.ams06prdstr12a.store.core.windows.net.
Name: dfs.ams06prdstr12a.store.core.windows.net
Address: 52.239.143.194
{code}
Which means that
# all abfs stores created with the same a/c share the same canonical name
# and on job submit, only one DT will be picked up.
# if the auth plugin for the FS uses its canonical service name for DT
retrieval, then every FS will pick up the relevant credentials, even if only
one FS instance has generated a DT
# and so if you call spark-submit for a job, and your cluster ifs is abfs://,
then the retrieved DT can be used for every abfs:// container, even without you
listing them in {{spark.yarn.access.hadoopFileSystems}}
What I'm going to do to retain this is have the bound token provider return
that canonical service name, or null, with the base/unbound value being null,
so falling back to the base class. With a test for this behaviour.
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755475#comment-16755475
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2
new + 4 unchanged - 0 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 23 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
19s{color} | {color:red} hadoop-tools_hadoop-azure generated 5 new + 0
unchanged - 0 fixed = 5 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 12s{color}
| {color:red} hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.azurebfs.extensions.TestDTManagerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12956801/HADOOP-16068-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c827a2bac7a3 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 04105bb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15870/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15870/artifact/out/whitespace-eol.txt
|
| javadoc |
[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS
[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750548#comment-16750548
]
Hadoop QA commented on HADOOP-16068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
1s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red} hadoop-tools/hadoop-azure generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-azure |
| | instanceof will always return true for all non-null values in
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(URI,
Configuration), since all
org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager are instances
of org.apache.hadoop.fs.azurebfs.extensions.CustomExtensionBinding At
AzureBlobFileSystem.java:for all non-null values in
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(URI,
Configuration), since all
org.apache.hadoop.fs.azurebfs.security.AbfsDelegationTokenManager are instances
of org.apache.hadoop.fs.azurebfs.extensions.CustomExtensionBinding At
AzureBlobFileSystem.java:[line 145] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16068 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12956050/HADOOP-16068-001.patch
|
|
