[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[
https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412342#comment-16412342
]
genericqa commented on HADOOP-15313:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
101 unchanged - 1 fixed = 101 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 24s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 37s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
43s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskChecker |
| | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15313 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915995/HADOOP-15313.03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6061ea9a27ea 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 647058e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
|
[jira] [Updated] (HADOOP-11123) Fix Java 9 incompatibilies in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-11123:
---
Summary: Fix Java 9 incompatibilies in Hadoop (was: Hadoop on Java 9)
> Fix Java 9 incompatibilies in Hadoop
>
>
> Key: HADOOP-11123
> URL: https://issues.apache.org/jira/browse/HADOOP-11123
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
> Environment: Java 9
>Reporter: Steve Loughran
>Priority: Critical
>
> JIRA to cover/track issues related to Hadoop on Java 9.
> Java 9 will have some significant changes —one of which is the removal of
> various {{com.sun}} classes. These removals need to be handled or Hadoop will
> not be able to run on a Java 9 JVM
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15338) Support Java 11 LTS in Hadoop
Akira Ajisaka created HADOOP-15338: -- Summary: Support Java 11 LTS in Hadoop Key: HADOOP-15338 URL: https://issues.apache.org/jira/browse/HADOOP-15338 Project: Hadoop Common Issue Type: Improvement Reporter: Akira Ajisaka Java 8 will be EoL during January 2019, so we need to support Java 11 LTS before the date. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15339: -- Status: Patch Available (was: Open) > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15339.001.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15339: -- Attachment: HADOOP-15339.001.patch > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15339.001.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15339: -- Component/s: common > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Bug > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15339: -- Issue Type: Improvement (was: Bug) > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410851#comment-16410851
]
Xiao Chen commented on HADOOP-14445:
Thanks for the review Rushabh, comments addressed in patch 8. Yes the duplicate
tokens works as the way you described, the new
{{testTokenCompatibilityOldRenewer}} verifies that. I also verified this in a
cluster around patch 4 (replace the jars everywhere except the RM host), if my
memory is correct... I'd be happy to verify the final patch again once all
comments are settled.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15313: --- Attachment: HADOOP-15313.01.patch > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410855#comment-16410855 ] Xiao Chen commented on HADOOP-15313: Patch 1 to close the providers. > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15313: --- Status: Patch Available (was: Open) > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410972#comment-16410972
]
genericqa commented on HADOOP-14445:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-common-project: The patch generated 3 new
+ 292 unchanged - 6 fixed = 295 total (was 298) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 18s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
35s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915822/HADOOP-14445.08.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 29ed9e71fdaa 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22c5ddb |
| maven | version: Apache Maven
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[
https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411022#comment-16411022
]
genericqa commented on HADOOP-15313:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project: The patch generated 1 new
+ 101 unchanged - 1 fixed = 102 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15313 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915824/HADOOP-15313.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2c52bb36325b 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22c5ddb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410900#comment-16410900
]
genericqa commented on HADOOP-14759:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 10
new + 5 unchanged - 0 fixed = 15 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 36s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14759 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915816/HADOOP-14759.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c4ecb435a0fa 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22c5ddb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14373/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14373/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14373/testReport/ |
| Max. process+thread count | 466 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U:
[jira] [Updated] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-11423: --- Summary: [Umbrella] Fix Java 10 incompatibilities in Hadoop (was: [Umbrella] Support Java 10 in Hadoop) > [Umbrella] Fix Java 10 incompatibilities in Hadoop > -- > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11423) [Umbrella] Support Java 10 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410860#comment-16410860 ] Akira Ajisaka commented on HADOOP-11423: Agreed. Updating the description. > [Umbrella] Support Java 10 in Hadoop > > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
Elek, Marton created HADOOP-15339: - Summary: Support additional key/value propereties in JMX bean registration Key: HADOOP-15339 URL: https://issues.apache.org/jira/browse/HADOOP-15339 Project: Hadoop Common Issue Type: Bug Reporter: Elek, Marton Assignee: Elek, Marton org.apache.hadoop.metrics2.util.MBeans.register is a utility function to register objects to the JMX registry with a given name prefix and name. JMX supports any additional key value pairs which could be part the the address of the jmx bean. For example: _java.lang:type=MemoryManager,name=CodeCacheManager_ Using this method we can query a group of mbeans, for example we can add the same tag to similar mbeans from namenode and datanode. This patch adds a small modification to support custom key value pairs and also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15340: -- Attachment: HADOOP-15340.001.patch > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
Elek, Marton created HADOOP-15340: - Summary: Fix the RPC server name usage to provide information about the metrics Key: HADOOP-15340 URL: https://issues.apache.org/jira/browse/HADOOP-15340 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.2.0 Reporter: Elek, Marton Assignee: Elek, Marton In case of multiple RPC servers in the same JVM it's hard to identify the metric data. The only available information as of now is the port number. Server name is also added in the constructor of Server.java but it's not used at all. This patch fix this behaviour: 1. The server name is saved to a field in Server.java (constructor signature is not changed) 2. ServerName is added as a tag to the metrics in RpcMetrics 3. The naming convention for the severs are fix. About 3: if the server name is not defined the current code tries to identify the name from the class name. Which is not always an easy task as in some cases the server has a protobuf generated dirty name which also could be an inner class. The patch also improved the detection of the name (if it's not defined). It's a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15340: -- Status: Patch Available (was: Open) > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15320) Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake
[
https://issues.apache.org/jira/browse/HADOOP-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411841#comment-16411841
]
Steve Loughran commented on HADOOP-15320:
-
OK, maybe I'm the confused one. Given it's working for S3a, if you're happy, so
am I
> Remove customized getFileBlockLocations for hadoop-azure and
> hadoop-azure-datalake
> --
>
> Key: HADOOP-15320
> URL: https://issues.apache.org/jira/browse/HADOOP-15320
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, fs/azure
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
> Attachments: HADOOP-15320.patch
>
>
> hadoop-azure and hadoop-azure-datalake have its own implementation of
> getFileBlockLocations(), which faked a list of artificial blocks based on the
> hard-coded block size. And each block has one host with name "localhost".
> Take a look at this code:
> [https://github.com/apache/hadoop/blob/release-2.9.0-RC3/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java#L3485]
> This is a unnecessary mock up for a "remote" file system to mimic HDFS. And
> the problem with this mock is that for large (~TB) files we generates lots of
> artificial blocks, and FileInputFormat.getSplits() is slow in calculating
> splits based on these blocks.
> We can safely remove this customized getFileBlockLocations() implementation,
> fall back to the default FileSystem.getFileBlockLocations() implementation,
> which is to return 1 block for any file with 1 host "localhost". Note that
> this doesn't mean we will create much less splits, because the number of
> splits is still limited by the blockSize in
> FileInputFormat.computeSplitSize():
> {code:java}
> return Math.max(minSize, Math.min(goalSize, blockSize));{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15307) Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
[
https://issues.apache.org/jira/browse/HADOOP-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15307:
Attachment: HADOOP-15307.001.patch
> Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
> ---
>
> Key: HADOOP-15307
> URL: https://issues.apache.org/jira/browse/HADOOP-15307
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
> Environment: CentOS 7.4, CDH5.13.1, Kerberized Hadoop cluster
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Labels: newbie
> Attachments: HADOOP-15307.001.patch
>
>
> When NFS gateway starts and if the portmapper request is denied by rpcbind
> for any reason (in our case, /etc/hosts.allow did not have the localhost),
> NFS gateway fails with the following obscure exception:
> {noformat}
> 2018-03-05 12:49:31,976 INFO org.apache.hadoop.oncrpc.SimpleUdpServer:
> Started listening to UDP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,988 INFO org.apache.hadoop.oncrpc.SimpleTcpServer:
> Started listening to TCP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,993 TRACE org.apache.hadoop.oncrpc.RpcCall:
> Xid:692394656, messageType:RPC_CALL, rpcVersion:2, program:10, version:2,
> procedure:1, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 2018-03-05 12:49:31,998 FATAL org.apache.hadoop.mount.MountdBase: Failed to
> start the server. Cause:
> java.lang.UnsupportedOperationException: Unsupported verifier flavorAUTH_SYS
> at
> org.apache.hadoop.oncrpc.security.Verifier.readFlavorAndVerifier(Verifier.java:45)
> at org.apache.hadoop.oncrpc.RpcDeniedReply.read(RpcDeniedReply.java:50)
> at org.apache.hadoop.oncrpc.RpcReply.read(RpcReply.java:67)
> at org.apache.hadoop.oncrpc.SimpleUdpClient.run(SimpleUdpClient.java:71)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:130)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:101)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:83)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> 2018-03-05 12:49:32,007 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1{noformat}
> Reading the code comment for class Verifier, I think this bug existed since
> its inception
> {code:java}
> /**
> * Base class for verifier. Currently our authentication only supports 3 types
> * of auth flavors: {@link RpcAuthInfo.AuthFlavor#AUTH_NONE}, {@link
> RpcAuthInfo.AuthFlavor#AUTH_SYS},
> * and {@link RpcAuthInfo.AuthFlavor#RPCSEC_GSS}. Thus for verifier we only
> need to handle
> * AUTH_NONE and RPCSEC_GSS
> */
> public abstract class Verifier extends RpcAuthInfo {{code}
> The verifier should also handle AUTH_SYS too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15307) Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
[
https://issues.apache.org/jira/browse/HADOOP-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15307:
Status: Patch Available (was: Open)
> Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
> ---
>
> Key: HADOOP-15307
> URL: https://issues.apache.org/jira/browse/HADOOP-15307
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
> Environment: CentOS 7.4, CDH5.13.1, Kerberized Hadoop cluster
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Labels: newbie
> Attachments: HADOOP-15307.001.patch
>
>
> When NFS gateway starts and if the portmapper request is denied by rpcbind
> for any reason (in our case, /etc/hosts.allow did not have the localhost),
> NFS gateway fails with the following obscure exception:
> {noformat}
> 2018-03-05 12:49:31,976 INFO org.apache.hadoop.oncrpc.SimpleUdpServer:
> Started listening to UDP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,988 INFO org.apache.hadoop.oncrpc.SimpleTcpServer:
> Started listening to TCP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,993 TRACE org.apache.hadoop.oncrpc.RpcCall:
> Xid:692394656, messageType:RPC_CALL, rpcVersion:2, program:10, version:2,
> procedure:1, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 2018-03-05 12:49:31,998 FATAL org.apache.hadoop.mount.MountdBase: Failed to
> start the server. Cause:
> java.lang.UnsupportedOperationException: Unsupported verifier flavorAUTH_SYS
> at
> org.apache.hadoop.oncrpc.security.Verifier.readFlavorAndVerifier(Verifier.java:45)
> at org.apache.hadoop.oncrpc.RpcDeniedReply.read(RpcDeniedReply.java:50)
> at org.apache.hadoop.oncrpc.RpcReply.read(RpcReply.java:67)
> at org.apache.hadoop.oncrpc.SimpleUdpClient.run(SimpleUdpClient.java:71)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:130)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:101)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:83)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> 2018-03-05 12:49:32,007 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1{noformat}
> Reading the code comment for class Verifier, I think this bug existed since
> its inception
> {code:java}
> /**
> * Base class for verifier. Currently our authentication only supports 3 types
> * of auth flavors: {@link RpcAuthInfo.AuthFlavor#AUTH_NONE}, {@link
> RpcAuthInfo.AuthFlavor#AUTH_SYS},
> * and {@link RpcAuthInfo.AuthFlavor#RPCSEC_GSS}. Thus for verifier we only
> need to handle
> * AUTH_NONE and RPCSEC_GSS
> */
> public abstract class Verifier extends RpcAuthInfo {{code}
> The verifier should also handle AUTH_SYS too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[
https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411837#comment-16411837
]
genericqa commented on HADOOP-15339:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 0 unchanged - 3 fixed = 0 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 18s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 35s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskChecker |
| | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15339 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915914/HADOOP-15339.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b0b638583d96 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6e31a09 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14379/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14379/testReport/ |
| Max. process+thread count | 1383 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[
https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411918#comment-16411918
]
genericqa commented on HADOOP-15313:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
101 unchanged - 1 fixed = 101 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 40s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
22s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
46s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15313 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915934/HADOOP-15313.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c9f358b8a06e 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6e31a09 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14380/testReport/ |
[jira] [Commented] (HADOOP-15320) Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake
[
https://issues.apache.org/jira/browse/HADOOP-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411827#comment-16411827
]
Chris Douglas commented on HADOOP-15320:
bq. I know s3 "appears" to work, but I'm not actually confident that everything
is getting the splits right there.
Me either, but 1.5h to generate synthetic splits is definitely wrong. If we
develop a new, best practice for object stores, then we can apply that across
the stores we support. The {{Blocklocations[]}} return type is pretty
restrictive, but we could probably do better.
bq. The one I want you look at is: Spark, CSV, multiGB: SPARK-22240 . That's
what's been niggling at me for a while.
Maybe I'm missing the bug. Block locations are hints for locality, not format
partitioning. In that JIRA: gzip is not splittable, so a single reader is
correct, absent some other preparation (saving the dictionary at offsets,
writing zero-length gzip files as split markers, etc.). In general, framework
parallelism should not rely exclusively on block locations...
> Remove customized getFileBlockLocations for hadoop-azure and
> hadoop-azure-datalake
> --
>
> Key: HADOOP-15320
> URL: https://issues.apache.org/jira/browse/HADOOP-15320
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, fs/azure
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
> Attachments: HADOOP-15320.patch
>
>
> hadoop-azure and hadoop-azure-datalake have its own implementation of
> getFileBlockLocations(), which faked a list of artificial blocks based on the
> hard-coded block size. And each block has one host with name "localhost".
> Take a look at this code:
> [https://github.com/apache/hadoop/blob/release-2.9.0-RC3/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java#L3485]
> This is a unnecessary mock up for a "remote" file system to mimic HDFS. And
> the problem with this mock is that for large (~TB) files we generates lots of
> artificial blocks, and FileInputFormat.getSplits() is slow in calculating
> splits based on these blocks.
> We can safely remove this customized getFileBlockLocations() implementation,
> fall back to the default FileSystem.getFileBlockLocations() implementation,
> which is to return 1 block for any file with 1 host "localhost". Note that
> this doesn't mean we will create much less splits, because the number of
> splits is still limited by the blockSize in
> FileInputFormat.computeSplitSize():
> {code:java}
> return Math.max(minSize, Math.min(goalSize, blockSize));{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411931#comment-16411931
]
Xiaoyu Yao commented on HADOOP-15295:
-
Thanks [~ajayydv] for working on this. The patch looks good to me overall. I
just have one question:
Configuration.java
Line 3187: can we keep the addTags() API simple as documented on line 3183
without the additional varagrs? {color:#00}Different components can add
customized tags via {color}HADOOP_CUSTOM_TAGS.
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15307) Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
[
https://issues.apache.org/jira/browse/HADOOP-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411999#comment-16411999
]
genericqa commented on HADOOP-15307:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 19s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15307 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915949/HADOOP-15307.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 51b45eba6f23 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6e31a09 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14381/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-nfs U:
hadoop-common-project/hadoop-nfs |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14381/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve NFS error handling:
[jira] [Updated] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HADOOP-15295:
Attachment: HADOOP-15295.002.patch
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412048#comment-16412048
]
Ajay Kumar commented on HADOOP-15295:
-
[~xyao], thanks for review. Removed var args in patch v2.
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[
https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412037#comment-16412037
]
Wei-Chiu Chuang commented on HADOOP-15313:
--
Thanks for the patch. Nice approach to capture them all.
I think you'll also need to add a provider into providersCreated in
createKMSClientProvider().
A few places where the provider is expliictly closed, like at line 475
{code:title=TestKMS#testStartStop}
testKp.close();
{code}
and at line 502, line 520, line 2197
> TestKMS should close providers
> --
>
> Key: HADOOP-15313
> URL: https://issues.apache.org/jira/browse/HADOOP-15313
> Project: Hadoop Common
> Issue Type: Test
> Components: kms, test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15313.01.patch, HADOOP-15313.02.patch
>
>
> During the review of HADOOP-14445, [~jojochuang] found that we key providers
> are not closed in tests. Details in [this
> comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824].
> We should investigate and handle that in all related tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15341) NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests fail on (overloaded?) client
Steve Loughran created HADOOP-15341: --- Summary: NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests fail on (overloaded?) client Key: HADOOP-15341 URL: https://issues.apache.org/jira/browse/HADOOP-15341 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.0.1 Environment: remote wasb client with IntelliJ idea also doing things like swapping Reporter: Steve Loughran Failures of NativeAzureFileSystemBaseTest tests for modified timestamps failing when the timestamp is 11 seconds out, rather than the 10s limit hard coded into the tests. Proposed: expand that range slightly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[
https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411345#comment-16411345
]
genericqa commented on HADOOP-15340:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 32m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 32m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 491 unchanged - 3 fixed = 491 total (was 494) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15340 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915851/HADOOP-15340.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a4940bf61e25 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 75fc05f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14376/testReport/ |
| Max. process+thread count | 1582 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14376/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix the RPC server name usage to provide information about
[jira] [Commented] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[
https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411350#comment-16411350
]
genericqa commented on HADOOP-15339:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 3 unchanged - 0 fixed = 12 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 18s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15339 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915847/HADOOP-15339.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cdbfcb24ff30 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 75fc05f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14377/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14377/testReport/ |
| Max. process+thread count | 1436 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14377/console |
| Powered by | Apache Yetus
[jira] [Commented] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411394#comment-16411394 ] Rushabh S Shah commented on HADOOP-11423: - If we agree that we will support only LTS releases then why java 10 ? It should be either java 8 or java 11. Did I miss something ? > [Umbrella] Fix Java 10 incompatibilities in Hadoop > -- > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-11423) [Umbrella] Fix Java 10 incompatibilities in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-11423: Comment: was deleted (was: If we agree that we will support only LTS releases then why java 10 ? It should be either java 8 or java 11. Did I miss something ?) > [Umbrella] Fix Java 10 incompatibilities in Hadoop > -- > > Key: HADOOP-11423 > URL: https://issues.apache.org/jira/browse/HADOOP-11423 > Project: Hadoop Common > Issue Type: New Feature >Reporter: sneaky >Priority: Minor > > Java 10 is coming quickly to various clusters. Making sure Hadoop seamlessly > works with Java 10 is important for the Apache community. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15341) NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests fail on (overloaded?) client
[ https://issues.apache.org/jira/browse/HADOOP-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15341: Environment: remote wasb client with IntelliJ idea also doing things like swapping; parallel test run enabled with threads = 5...though the test which failed was in the sequential phase (was: remote wasb client with IntelliJ idea also doing things like swapping) > NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests > fail on (overloaded?) client > --- > > Key: HADOOP-15341 > URL: https://issues.apache.org/jira/browse/HADOOP-15341 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.1 > Environment: remote wasb client with IntelliJ idea also doing things > like swapping; parallel test run enabled with threads = 5...though the test > which failed was in the sequential phase >Reporter: Steve Loughran >Priority: Minor > > Failures of NativeAzureFileSystemBaseTest tests for modified timestamps > failing when the timestamp is 11 seconds out, rather than the 10s limit hard > coded into the tests. > Proposed: expand that range slightly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15341) NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests fail on (overloaded?) client
[
https://issues.apache.org/jira/browse/HADOOP-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411338#comment-16411338
]
Steve Loughran commented on HADOOP-15341:
-
{code}
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 261.714 sec -
in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads
Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
Tests run: 52, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 268.819 sec
<<< FAILURE! - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive
testModifiedTimeForFolder(org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive)
Time elapsed: 0.635 sec <<< FAILURE!
java.lang.AssertionError: Modification time Fri Mar 23 12:52:20 GMT 2018 is not
close to now: Fri Mar 23 12:52:09 GMT 2018
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystemBaseTest.testModifiedTime(NativeAzureFileSystemBaseTest.java:1492)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystemBaseTest.testModifiedTimeForFolder(NativeAzureFileSystemBaseTest.java:633)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
testFolderLastModifiedTime(org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive)
Time elapsed: 6.305 sec <<< FAILURE!
java.lang.AssertionError: Modification time Fri Mar 23 12:52:35 GMT 2018 is not
close to now: Fri Mar 23 12:52:24 GMT 2018
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystemBaseTest.testModifiedTime(NativeAzureFileSystemBaseTest.java:1492)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystemBaseTest.testFolderLastModifiedTime(NativeAzureFileSystemBaseTest.java:651)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> NativeAzureFileSystemBaseTest.testModifiedTimeForFolder modification tests
> fail on (overloaded?) client
> ---
>
> Key: HADOOP-15341
> URL: https://issues.apache.org/jira/browse/HADOOP-15341
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.0.1
> Environment: remote wasb client with IntelliJ idea also doing things
> like swapping
>Reporter: Steve Loughran
>Priority: Minor
>
> Failures of NativeAzureFileSystemBaseTest tests for modified timestamps
> failing when the timestamp is 11 seconds out, rather than the 10s limit hard
> coded into the tests.
> Proposed: expand that range slightly
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-15188) azure datalake AzureADAuthenticator failing, no error info provided
[
https://issues.apache.org/jira/browse/HADOOP-15188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411347#comment-16411347
]
Steve Loughran commented on HADOOP-15188:
-
Update on this : you see it if there is a proxy in the way and the returned
message is not JSON
# the ADL SDK should be checking content-type for being JSON and content length
>0 before trying to parse the JSON; reporting the failures differently.
# reporting proxy settings at debug level would also be useful.
see also [[https://github.com/steveloughran/cloudstore]], which was used to
track down this issue
> azure datalake AzureADAuthenticator failing, no error info provided
> ---
>
> Key: HADOOP-15188
> URL: https://issues.apache.org/jira/browse/HADOOP-15188
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Steve Loughran
>Assignee: Atul Sikaria
>Priority: Major
>
> Get a failure in ADLS client, but nothing useful in terms of failure
> description
> {code}
> DEBUG oauth2.AzureADAuthenticator: AADToken: starting to fetch token using
> client creds for client ID
> DEBUG store.HttpTransport:
> HTTPRequest,Failed,cReqId:,lat:127370,err:HTTP0(null),Reqlen:0,Resplen:0,token_ns:,sReqId:null,path:,qp:op=GETFILESTATUS=true=2016-11-01
> {code}
> so: we had a failure but the response code is 0, error(null); "something
> happened but we don't know what"
> Looks like this log message is in the ADLS SDK, and can be translated like
> this.
> {code}
> String logline =
> "HTTPRequest," + outcome +
> ",cReqId:" + opts.requestid +
> ",lat:" + Long.toString(resp.lastCallLatency) +
> ",err:" + error +
> ",Reqlen:" + length +
> ",Resplen:" + respLength +
> ",token_ns:" + Long.toString(resp.tokenAcquisitionLatency) +
> ",sReqId:" + resp.requestId +
> ",path:" + path +
> ",qp:" + queryParams.serialize();
> {code}
> It looks like whatever code tries to parse the JSON response from the OAuth
> service couldn't make sense of the response, and we end up with nothing back.
> Not sure what can be done in hadoop to handle this, except maybe provide more
> diags on request failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14759:
Status: Open (was: Patch Available)
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14759.001.patch, HADOOP-14759.002.patch
>
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14759:
Attachment: HADOOP-14759.002.patch
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14759.001.patch, HADOOP-14759.002.patch
>
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14759:
Status: Patch Available (was: Open)
Fix checkstyle and whitespace issues
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14759.001.patch, HADOOP-14759.002.patch
>
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11219) Upgrade to netty 4
[ https://issues.apache.org/jira/browse/HADOOP-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411538#comment-16411538 ] Jonathan Eagles commented on HADOOP-11219: -- In Tez, we provide a Tez Shuffle Handler that is similar to MR Shuffle Handler with some added feature support. There may be an interaction here that needs to be tested and verified. I'd be happy to verify once there is a patch available. > Upgrade to netty 4 > -- > > Key: HADOOP-11219 > URL: https://issues.apache.org/jira/browse/HADOOP-11219 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Major > > This is an umbrella jira to track the effort of upgrading to Netty 4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411719#comment-16411719
]
Chris Douglas commented on HADOOP-14600:
bq. Without realizing you had already fixed this issue, I put a patch up in
HADOOP-15337 that shows the alternative implementation approach. Sorry again
for being so late to the conversation, not intending to step on toes.
No worries. If the approach in HADOOP-15337 is more portable, performs
comparably, and is equivalent, then let's take the simpler approach. Where
applicable, do the unit tests on this issue pass in HADOOP-15337, on
Linux/MacOS/Windows?
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch,
> HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch,
> HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch,
> HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java,
> command_line_test_result__linux.txt, command_line_test_result__windows.txt
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15320) Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake
[
https://issues.apache.org/jira/browse/HADOOP-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411716#comment-16411716
]
Steve Loughran commented on HADOOP-15320:
-
I know s3 "appears" to work, but I'm not actually confident that everything is
getting the splits right there.
The one I want you look at is: Spark, CSV, multiGB: SPARK-22240 . That's what's
been niggling at me for a while.
> Remove customized getFileBlockLocations for hadoop-azure and
> hadoop-azure-datalake
> --
>
> Key: HADOOP-15320
> URL: https://issues.apache.org/jira/browse/HADOOP-15320
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, fs/azure
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
>Priority: Major
> Attachments: HADOOP-15320.patch
>
>
> hadoop-azure and hadoop-azure-datalake have its own implementation of
> getFileBlockLocations(), which faked a list of artificial blocks based on the
> hard-coded block size. And each block has one host with name "localhost".
> Take a look at this code:
> [https://github.com/apache/hadoop/blob/release-2.9.0-RC3/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java#L3485]
> This is a unnecessary mock up for a "remote" file system to mimic HDFS. And
> the problem with this mock is that for large (~TB) files we generates lots of
> artificial blocks, and FileInputFormat.getSplits() is slow in calculating
> splits based on these blocks.
> We can safely remove this customized getFileBlockLocations() implementation,
> fall back to the default FileSystem.getFileBlockLocations() implementation,
> which is to return 1 block for any file with 1 host "localhost". Note that
> this doesn't mean we will create much less splits, because the number of
> splits is still limited by the blockSize in
> FileInputFormat.computeSplitSize():
> {code:java}
> return Math.max(minSize, Math.min(goalSize, blockSize));{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15307) Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
[
https://issues.apache.org/jira/browse/HADOOP-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-15307:
---
Assignee: Gabor Bota
> Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
> ---
>
> Key: HADOOP-15307
> URL: https://issues.apache.org/jira/browse/HADOOP-15307
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
> Environment: CentOS 7.4, CDH5.13.1, Kerberized Hadoop cluster
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Labels: newbie
>
> When NFS gateway starts and if the portmapper request is denied by rpcbind
> for any reason (in our case, /etc/hosts.allow did not have the localhost),
> NFS gateway fails with the following obscure exception:
> {noformat}
> 2018-03-05 12:49:31,976 INFO org.apache.hadoop.oncrpc.SimpleUdpServer:
> Started listening to UDP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,988 INFO org.apache.hadoop.oncrpc.SimpleTcpServer:
> Started listening to TCP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,993 TRACE org.apache.hadoop.oncrpc.RpcCall:
> Xid:692394656, messageType:RPC_CALL, rpcVersion:2, program:10, version:2,
> procedure:1, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 2018-03-05 12:49:31,998 FATAL org.apache.hadoop.mount.MountdBase: Failed to
> start the server. Cause:
> java.lang.UnsupportedOperationException: Unsupported verifier flavorAUTH_SYS
> at
> org.apache.hadoop.oncrpc.security.Verifier.readFlavorAndVerifier(Verifier.java:45)
> at org.apache.hadoop.oncrpc.RpcDeniedReply.read(RpcDeniedReply.java:50)
> at org.apache.hadoop.oncrpc.RpcReply.read(RpcReply.java:67)
> at org.apache.hadoop.oncrpc.SimpleUdpClient.run(SimpleUdpClient.java:71)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:130)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:101)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:83)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> 2018-03-05 12:49:32,007 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1{noformat}
> Reading the code comment for class Verifier, I think this bug existed since
> its inception
> {code:java}
> /**
> * Base class for verifier. Currently our authentication only supports 3 types
> * of auth flavors: {@link RpcAuthInfo.AuthFlavor#AUTH_NONE}, {@link
> RpcAuthInfo.AuthFlavor#AUTH_SYS},
> * and {@link RpcAuthInfo.AuthFlavor#RPCSEC_GSS}. Thus for verifier we only
> need to handle
> * AUTH_NONE and RPCSEC_GSS
> */
> public abstract class Verifier extends RpcAuthInfo {{code}
> The verifier should also handle AUTH_SYS too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15299) Bump Hadoop's Jackson 2 dependency 2.9.x
[ https://issues.apache.org/jira/browse/HADOOP-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412302#comment-16412302 ] Daniel Templeton commented on HADOOP-15299: --- The change looks safe enough, as long as you've tested it thoroughly, which it sounds like you have. Given that it closes out some CVEs, I'd say we should get it in. +1 > Bump Hadoop's Jackson 2 dependency 2.9.x > > > Key: HADOOP-15299 > URL: https://issues.apache.org/jira/browse/HADOOP-15299 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15299.001.patch > > > There are a few new CVEs open against Jackson 2.7.x. It doesn't (necessarily) > mean Hadoop is vulnerable to the attack - I don't know that it is, but fixes > were released for Jackson 2.8.x and 2.9.x but not 2.7.x (which we're on). We > shouldn't be on an unmaintained line, regardless. HBase is already on 2.9.x, > we have a shaded client now, the API changes are relatively minor and so far > in my testing I haven't seen any problems. I think many of our usual reasons > to hesitate upgrading this dependency don't apply. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412309#comment-16412309
]
genericqa commented on HADOOP-15295:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 4s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 6s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestTrash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915979/HADOOP-15295.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 20a2c63f6449 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 647058e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14382/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14382/testReport/ |
| Max. process+thread count | 1441 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14382/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Created] (HADOOP-15342) Update ADLS connector to use the current SDK version (2.2.7)
Atul Sikaria created HADOOP-15342: - Summary: Update ADLS connector to use the current SDK version (2.2.7) Key: HADOOP-15342 URL: https://issues.apache.org/jira/browse/HADOOP-15342 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Reporter: Atul Sikaria Assignee: Atul Sikaria Updating the ADLS SDK connector to use the current version of the ADLS SDK (2.2.7). Changelist is here: [https://github.com/Azure/azure-data-lake-store-java/blob/sdk2.2/CHANGES.md] Short summary of what matters: Change to the MSI token acquisition interface required by the change in REST interface to Azure ActiveDirecotry's VM MSI interface, and improved diagnostics in the SDK for token acquisition failures (better exception message and log message). The diagnostics was requested in HADOOP-15188. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15188) azure datalake AzureADAuthenticator failing, no error info provided
[
https://issues.apache.org/jira/browse/HADOOP-15188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412311#comment-16412311
]
Atul Sikaria commented on HADOOP-15188:
---
New SDK with requested changes has been published to Maven Central, the
corresponding update to the ADLS connector is tracked in HADOOP-15342.
> azure datalake AzureADAuthenticator failing, no error info provided
> ---
>
> Key: HADOOP-15188
> URL: https://issues.apache.org/jira/browse/HADOOP-15188
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Steve Loughran
>Assignee: Atul Sikaria
>Priority: Major
>
> Get a failure in ADLS client, but nothing useful in terms of failure
> description
> {code}
> DEBUG oauth2.AzureADAuthenticator: AADToken: starting to fetch token using
> client creds for client ID
> DEBUG store.HttpTransport:
> HTTPRequest,Failed,cReqId:,lat:127370,err:HTTP0(null),Reqlen:0,Resplen:0,token_ns:,sReqId:null,path:,qp:op=GETFILESTATUS=true=2016-11-01
> {code}
> so: we had a failure but the response code is 0, error(null); "something
> happened but we don't know what"
> Looks like this log message is in the ADLS SDK, and can be translated like
> this.
> {code}
> String logline =
> "HTTPRequest," + outcome +
> ",cReqId:" + opts.requestid +
> ",lat:" + Long.toString(resp.lastCallLatency) +
> ",err:" + error +
> ",Reqlen:" + length +
> ",Resplen:" + respLength +
> ",token_ns:" + Long.toString(resp.tokenAcquisitionLatency) +
> ",sReqId:" + resp.requestId +
> ",path:" + path +
> ",qp:" + queryParams.serialize();
> {code}
> It looks like whatever code tries to parse the JSON response from the OAuth
> service couldn't make sense of the response, and we end up with nothing back.
> Not sure what can be done in hadoop to handle this, except maybe provide more
> diags on request failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412180#comment-16412180
]
Xiaoyu Yao commented on HADOOP-15295:
-
Thanks [~ajayydv] for the update. Patch v2 looks good to me. +1 pending Jenkins.
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15313: --- Attachment: HADOOP-15313.03.patch > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch, HADOOP-15313.02.patch, > HADOOP-15313.03.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412170#comment-16412170 ] Xiao Chen commented on HADOOP-15313: Thanks for the review [~jojochuang], good catch. Handled createKMSCP, and removed the explicit close on 502 / 520. We still need the explicit close at 475 / 2197 to verify the close method itself doesn't leak threads, so left these 2 as-is. (Added a comment to 475 to make this clearer) > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch, HADOOP-15313.02.patch, > HADOOP-15313.03.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412174#comment-16412174 ] Xiaoyu Yao commented on HADOOP-15340: - Thanks [~elek] for reporting the issue and posting the fix. The patch looks good to me overall. Just have two minor suggestions: RPC.java Line 816-820: can you move comments into a javadoc with examples illustrating the expected behavior If I understand correctly, serverNameFromClass tries to get meaningful server name based on class name passed in. The first two cases should work with getSimpleName. 0. Simple class works the same as getSimpleName(). 1. If it is an embedded class, returns the inner class name (same as the getSimpleName()) E.g., o.a.h.h.proto.classA$classB$classC then return classC Cases 2-4 are where getSimpleNmae() returns either '' or null. We should document the expected behavior: 2. If it is an Object created from a class factory E.g., org.apache.hadoop.ipc.TestRPC$TestClass$2 serverNameFromClass returns parent class TestClass. 3. If it is an anonymous class E.g., 'org.apache.hadoop.ipc.TestRPC$10' serverNameFromClass returns parent class TestRPC. 4. If it is a lambda runnable/consumer, etc. class E.g., 'org.apache.hadoop.ipc.TestRPC$$Lambda$1/1360767589' serverNameFromClass returns "Lambda", should we return parent class here instead of Lambda? Line 826: NIT: pattern can be moved to static final member of the abstract class. > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412183#comment-16412183 ] Wei-Chiu Chuang commented on HADOOP-15313: -- +1 pending Jenkins. Thank you. > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch, HADOOP-15313.02.patch, > HADOOP-15313.03.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412226#comment-16412226 ] Xiaoyu Yao commented on HADOOP-15339: - Thanks [~elek] for reporting the issue and posting the fix. The patch looks good to me. Just one suggestion: MBeans.java Line 55: unused empty private constructor, can we remove it? Line 90: can we add a precondition check to ensure properties map is not null as getMBeanName cannot handle that. > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15339.001.patch, HADOOP-15339.002.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412235#comment-16412235
]
Aaron Fabbri commented on HADOOP-14759:
---
Thanks for the patch [~gabor.bota]. Looks pretty good. A couple of comments:
{noformat}
@Override
- @Retries.OnceRaw("once(batchWrite)")
public void prune(long modTime) throws IOException {
{noformat}
Looks like an accidental deletion of the annotation, should probably add it
back.
{noformat}
+ // remove the protocol from path string to get parent
+ // should keep one starting slash
+ String keyPrefix = paths.get(0).replaceFirst("s3a:/", "");
{noformat}
You can probably use PathMetadataDynamoDBTranslation#pathToParentKey() here
instead.
Also, do we need to update any of the docs in
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws?
{quote}
Tests ran on us-west-2 successfully.
{quote}
Thank you for declaring where you ran tests. This included both unit (mvn
test) and integration tests (mvn verify), right?
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-14759.001.patch, HADOOP-14759.002.patch
>
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15338) Support Java 11 LTS in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15338: --- Description: Oracle JDK 8 will be EoL during January 2019, and RedHat will end support for OpenJDK 8 in October 2020 ([https://access.redhat.com/articles/1299013]), so we need to support Java 11 LTS at least before October 2020. was:Java 8 will be EoL during January 2019, so we need to support Java 11 LTS before the date. > Support Java 11 LTS in Hadoop > - > > Key: HADOOP-15338 > URL: https://issues.apache.org/jira/browse/HADOOP-15338 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > > Oracle JDK 8 will be EoL during January 2019, and RedHat will end support for > OpenJDK 8 in October 2020 ([https://access.redhat.com/articles/1299013]), so > we need to support Java 11 LTS at least before October 2020. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411641#comment-16411641
]
genericqa commented on HADOOP-14759:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14759 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915910/HADOOP-14759.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 090c4124a2e2 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 75fc05f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14378/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14378/testReport/ |
| Max. process+thread count | 463 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14378/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Updated] (HADOOP-15313) TestKMS should close providers
[ https://issues.apache.org/jira/browse/HADOOP-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15313: --- Attachment: HADOOP-15313.02.patch > TestKMS should close providers > -- > > Key: HADOOP-15313 > URL: https://issues.apache.org/jira/browse/HADOOP-15313 > Project: Hadoop Common > Issue Type: Test > Components: kms, test >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15313.01.patch, HADOOP-15313.02.patch > > > During the review of HADOOP-14445, [~jojochuang] found that we key providers > are not closed in tests. Details in [this > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16397824=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16397824]. > We should investigate and handle that in all related tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14435) TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
[
https://issues.apache.org/jira/browse/HADOOP-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411348#comment-16411348
]
Steve Loughran commented on HADOOP-14435:
-
I've just seen this too on 3.0.1RC0, where things were all working before.
Interesting.
> TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
> --
>
> Key: HADOOP-14435
> URL: https://issues.apache.org/jira/browse/HADOOP-14435
> Project: Hadoop Common
> Issue Type: Task
> Components: documentation, fs/adl
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Saw the following assertion failure in branch-2 and trunk:
> {noformat}
> Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.189 sec
> <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive
> testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive)
> Time elapsed: 0.71 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<461> but was:<456>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:219)
> at junit.framework.Assert.assertEquals(Assert.java:226)
> at junit.framework.TestCase.assertEquals(TestCase.java:392)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:242)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive.runTest(TestAdlFileSystemContractLive.java:59)
> Results :
> Failed tests:
>
> TestAdlFileSystemContractLive.runTest:59->FileSystemContractBaseTest.testMkdirsWithUmask:242
> expected:<461> but was:<456>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15337) RawLocalFileSystem file status permissions can avoid shelling out in some cases
[ https://issues.apache.org/jira/browse/HADOOP-15337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-15337: - Attachment: HADOOP-15337.001.patch > RawLocalFileSystem file status permissions can avoid shelling out in some > cases > --- > > Key: HADOOP-15337 > URL: https://issues.apache.org/jira/browse/HADOOP-15337 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: HADOOP-15337.001.patch > > > While investigating YARN-8054, it was noticed that getting file permissions > for RawLocalFileSystem can fail by having too many files open. Upon > inspection this happens when getting permissions by launching a shell program > (ls -ld on linux) and parsing the results. With the introduction of java 7 > posix file systems can accurately get file permissions without launching a > shell program. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411625#comment-16411625 ] Rushabh S Shah commented on HADOOP-15336: - Also was client 2.7 or 3.2 ? and same for server. > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.2 > - > > Key: HADOOP-15336 > URL: https://issues.apache.org/jira/browse/HADOOP-15336 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Major > Labels: backward-incompatible, common > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411548#comment-16411548 ] Elek, Marton commented on HADOOP-15339: --- New version is uploaded with checkstyle fixes. > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15339.001.patch, HADOOP-15339.002.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15339) Support additional key/value propereties in JMX bean registration
[ https://issues.apache.org/jira/browse/HADOOP-15339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15339: -- Attachment: HADOOP-15339.002.patch > Support additional key/value propereties in JMX bean registration > - > > Key: HADOOP-15339 > URL: https://issues.apache.org/jira/browse/HADOOP-15339 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15339.001.patch, HADOOP-15339.002.patch > > > org.apache.hadoop.metrics2.util.MBeans.register is a utility function to > register objects to the JMX registry with a given name prefix and name. > JMX supports any additional key value pairs which could be part the the > address of the jmx bean. For example: > _java.lang:type=MemoryManager,name=CodeCacheManager_ > Using this method we can query a group of mbeans, for example we can add the > same tag to similar mbeans from namenode and datanode. > This patch adds a small modification to support custom key value pairs and > also introduce a new unit test for MBeans utility which was missing until now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411586#comment-16411586
]
Jonathan Eagles commented on HADOOP-14600:
--
[~myapachejira] Was taking a look at this exact bug I found this week while
running 2.8. Filed HADOOP-15337 to fix, but this is likely the same exactly.
It would be great to have this feature fix in 2.8, but I see this JIRA took a
vastly different approach that seems more work to backport. Did you compare the
posix file system approach implementation approach for speeds and support?
Without realizing you had already fixed this issue, I put a patch up in
HADOOP-15337 that shows the alternative implementation approach. Sorry again
for being so late to the conversation, not intending to step on toes.
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
>
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
>Reporter: Steve Loughran
>Assignee: Ping Liu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch,
> HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch,
> HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch,
> HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java,
> command_line_test_result__linux.txt, command_line_test_result__windows.txt
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15336) NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.2
[ https://issues.apache.org/jira/browse/HADOOP-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411600#comment-16411600 ] Jason Lowe commented on HADOOP-15336: - Could you elaborate in the description on the nature of the NPE? A sample stacktrace would be immensely helpful here. > NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication > between 2.7 and 3.2 > - > > Key: HADOOP-15336 > URL: https://issues.apache.org/jira/browse/HADOOP-15336 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0 >Reporter: Sherwood Zheng >Assignee: Sherwood Zheng >Priority: Major > Labels: backward-incompatible, common > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-15295:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
[~ajayydv] Thank you for the contribution. I have committed this to the trunk.
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15317) Improve NetworkTopology chooseRandom's loop
[
https://issues.apache.org/jira/browse/HADOOP-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15317:
---
Attachment: HADOOP-15317.03.patch
> Improve NetworkTopology chooseRandom's loop
> ---
>
> Key: HADOOP-15317
> URL: https://issues.apache.org/jira/browse/HADOOP-15317
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15317.01.patch, HADOOP-15317.02.patch,
> HADOOP-15317.03.patch
>
>
> Recently we found a postmortem case where the ANN seems to be in an infinite
> loop. From the logs it seems it just went through a rolling restart, and DNs
> are getting registered.
> Later the NN become unresponsive, and from the stacktrace it's inside a
> do-while loop inside {{NetworkTopology#chooseRandom}} - part of what's done
> in HDFS-10320.
> Going through the code and logs I'm not able to come up with any theory
> (thought about incorrect locking, or the Node object being modified outside
> of NetworkTopology, both seem impossible) why this is happening, but we
> should eliminate this loop.
> stacktrace:
> {noformat}
> Stack:
> java.util.HashMap.hash(HashMap.java:338)
> java.util.HashMap.containsKey(HashMap.java:595)
> java.util.HashSet.contains(HashSet.java:203)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:786)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:732)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:757)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:692)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:573)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:461)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:368)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:243)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:115)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode(BlockManager.java:1596)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3599)
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:717)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15295) Remove redundant logging related to tags from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412422#comment-16412422
]
Hudson commented on HADOOP-15295:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13874 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13874/])
HADOOP-15295. Remove redundant logging related to tags from (aengineer: rev
28790b81ecb719ac26a01ad0c400ee5f1c29ccf8)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
> Remove redundant logging related to tags from Configuration
> ---
>
> Key: HADOOP-15295
> URL: https://issues.apache.org/jira/browse/HADOOP-15295
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15295.000.patch, HADOOP-15295.001.patch,
> HADOOP-15295.002.patch
>
>
> Remove redundant logging related to tags from Configuration.
> {code}
> 2018-03-06 18:55:46,164 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,237 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,249 INFO conf.Configuration: Removed undeclared tags:
> 2018-03-06 18:55:46,256 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15317) Improve NetworkTopology chooseRandom's loop
[
https://issues.apache.org/jira/browse/HADOOP-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412427#comment-16412427
]
Xiao Chen commented on HADOOP-15317:
Thanks for the review [~ajayydv]. Comments addressed, and added a special case
to handle _the_ best case scenario as an improvement. Notably, this may still
end up with more iterations than the old simple stupid way, for scenarios like
when the node would have been chosen by the 2nd random choose in old logic.
> Improve NetworkTopology chooseRandom's loop
> ---
>
> Key: HADOOP-15317
> URL: https://issues.apache.org/jira/browse/HADOOP-15317
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15317.01.patch, HADOOP-15317.02.patch,
> HADOOP-15317.03.patch
>
>
> Recently we found a postmortem case where the ANN seems to be in an infinite
> loop. From the logs it seems it just went through a rolling restart, and DNs
> are getting registered.
> Later the NN become unresponsive, and from the stacktrace it's inside a
> do-while loop inside {{NetworkTopology#chooseRandom}} - part of what's done
> in HDFS-10320.
> Going through the code and logs I'm not able to come up with any theory
> (thought about incorrect locking, or the Node object being modified outside
> of NetworkTopology, both seem impossible) why this is happening, but we
> should eliminate this loop.
> stacktrace:
> {noformat}
> Stack:
> java.util.HashMap.hash(HashMap.java:338)
> java.util.HashMap.containsKey(HashMap.java:595)
> java.util.HashSet.contains(HashSet.java:203)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:786)
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:732)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:757)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:692)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:573)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:461)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:368)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:243)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:115)
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode(BlockManager.java:1596)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3599)
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:717)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
