[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675938#comment-15675938
]
Xiao Chen commented on HADOOP-13805:
Hm, clearly unit test caught me. Let me look more into this...
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13805.01.patch, HADOOP-13805.02.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13660) Upgrade commons-configuration version
[ https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675782#comment-15675782 ] Hudson commented on HADOOP-13660: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10861 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10861/]) HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean (weichiu: rev c0b1a44f6c6e6f9e4ac5cecea0d4a50e237a4c9c) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/StatsDSink.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java * (edit) hadoop-project/pom.xml * (edit) hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/ganglia/TestGangliaSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/GraphiteSink.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigUtil.java * (edit) hadoop-tools/hadoop-kafka/src/test/java/org/apache/hadoop/metrics2/impl/TestKafkaMetrics.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsConfig.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/GangliaSink30.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/ConfigBuilder.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsCollectorImpl.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSinkTestBase.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestRollingFileSystemSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/filter/AbstractPatternFilter.java * (edit) hadoop-tools/hadoop-kafka/src/main/java/org/apache/hadoop/metrics2/sink/KafkaSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/FileSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/AbstractGangliaSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java * (edit) hadoop-tools/hadoop-azure/pom.xml * (edit) hadoop-common-project/hadoop-common/pom.xml * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/filter/TestPatternFilter.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/PositionStripeReader.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/RollingFileSystemSink.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeFSDataSetSink.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsPlugin.java > Upgrade commons-configuration version > - > > Key: HADOOP-13660 > URL: https://issues.apache.org/jira/browse/HADOOP-13660 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13660-configuration2.001.patch, > HADOOP-13660.001.patch, HADOOP-13660.002.patch, HADOOP-13660.003.patch, > HADOOP-13660.004.patch, HADOOP-13660.005.patch, HADOOP-13660.006.patch, > HADOOP-13660.007.patch > > > We're currently pulling in version 1.6 - I think we should upgrade to the > latest 1.10. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13660) Upgrade commons-configuration version
[ https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675768#comment-15675768 ] Wei-Chiu Chuang commented on HADOOP-13660: -- Committed the 007 patch to trunk. Thanks [~mackrorysd] for the great contribution! > Upgrade commons-configuration version > - > > Key: HADOOP-13660 > URL: https://issues.apache.org/jira/browse/HADOOP-13660 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13660-configuration2.001.patch, > HADOOP-13660.001.patch, HADOOP-13660.002.patch, HADOOP-13660.003.patch, > HADOOP-13660.004.patch, HADOOP-13660.005.patch, HADOOP-13660.006.patch, > HADOOP-13660.007.patch > > > We're currently pulling in version 1.6 - I think we should upgrade to the > latest 1.10. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13660) Upgrade commons-configuration version
[ https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13660: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 Release Note: Bump commons-configuration version from 1.6 to 2.1 Status: Resolved (was: Patch Available) > Upgrade commons-configuration version > - > > Key: HADOOP-13660 > URL: https://issues.apache.org/jira/browse/HADOOP-13660 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13660-configuration2.001.patch, > HADOOP-13660.001.patch, HADOOP-13660.002.patch, HADOOP-13660.003.patch, > HADOOP-13660.004.patch, HADOOP-13660.005.patch, HADOOP-13660.006.patch, > HADOOP-13660.007.patch > > > We're currently pulling in version 1.6 - I think we should upgrade to the > latest 1.10. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675717#comment-15675717
]
Hadoop QA commented on HADOOP-13805:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 57s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestUGIWithMiniKdc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13805 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839492/HADOOP-13805.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux eab41fcf714b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 140b993 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11096/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11096/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11096/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
>
[jira] [Updated] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13651: -- Status: Patch Available (was: Open) > S3Guard: S3AFileSystem Integration with MetadataStore > - > > Key: HADOOP-13651 > URL: https://issues.apache.org/jira/browse/HADOOP-13651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-13651-HADOOP-13345.001.patch, > HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch, > HADOOP-13651-HADOOP-13345.004.patch, HADOOP-13651-HADOOP-13345.005.patch, > HADOOP-13651-HADOOP-13345.006.patch, v6-test-report.txt > > > Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata > consistency and caching. > Implementation should have minimal overhead when no MetadataStore is > configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13651: -- Status: Open (was: Patch Available) > S3Guard: S3AFileSystem Integration with MetadataStore > - > > Key: HADOOP-13651 > URL: https://issues.apache.org/jira/browse/HADOOP-13651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-13651-HADOOP-13345.001.patch, > HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch, > HADOOP-13651-HADOOP-13345.004.patch, HADOOP-13651-HADOOP-13345.005.patch, > HADOOP-13651-HADOOP-13345.006.patch, v6-test-report.txt > > > Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata > consistency and caching. > Implementation should have minimal overhead when no MetadataStore is > configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675673#comment-15675673
]
Hadoop QA commented on HADOOP-13449:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 55s{color}
| {color:red} root generated 5 new + 695 unchanged - 0 fixed = 700 total (was
695) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 30s{color} | {color:orange} root: The patch generated 10 new + 4 unchanged -
0 fixed = 14 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13449 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839489/HADOOP-13449-HADOOP-13345.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 9c2561317d10 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
[jira] [Updated] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13805:
---
Status: Patch Available (was: Open)
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13805.01.patch, HADOOP-13805.02.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13805:
---
Attachment: HADOOP-13805.02.patch
Thanks [~tucu00] for the review. I thought more about this, and feels like we
should add an orthogonal parameter to control whether to relogin/spawn renew
thread or not.
Patch 2 attached, I believe this should take care of both this and HADOOP-13807
correctly.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13805.01.patch, HADOOP-13805.02.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13449:
---
Attachment: HADOOP-13449-HADOOP-13345.005.patch
Thanks for the discussion, [~fabbri]. That's very helpful.
{quote}
for v1, you could always return authoritative = false.
{quote}
Yes, it's the current patch. Let's address this as a follow-up JIRA after the
[HADOOP-13651] and this both be committed.
{quote}
The interface allows any of these behaviors The filesystem is responsible
for ensuring that the delete to /a must be recursive since it is not empty.
MetadataStore explicitly does not do that.
{quote}
Agreed. For example, {{delete(path)}} does not check the directory path being
empty.
{quote}
You either have to (A) pay money to store an extra copy of your metadata
forever, or (B) spend money and time hydrating the MetadataStore each time you
start a cluster.
{quote}
The metadata size is considered small and the price of DDB storage is low
comparing with read/write operations pricing. If I have to choose, (A) makes
more sense.
{quote}
and we don't assume everything is always in DynamoDB, it makes recovery much
easier
{quote}
That's very valid. Altering S3 and MetadataStore is not atomic.
{quote}
The other concern is that I just don't understand why you would want to do the
preloading.
{quote}
You mean import? I suppose not. For read/write existing s3 buckets, importing
the structure first seems a prerequisite unless we assume it
discovers/converges fast or we reach little consistency.
I guess you mean the constrictions on the pre-creating parent directories. I
re-read the design doc and [HADOOP-13651] patch, and think you made a good
point about this. Let S3AFileSystem ensure the contract.
Moreover, I now think storing the is_empty bit in DynamoDB is not ideal.
Maintaining it needs non-trivial effort and it's easy to make it wrong. Perhaps
we can query via parent directories as HASH key when we need this information.
This is non-trivial either; I'll think about this as my next work. We can
either fix this in next patch, or I'll work on a follow-up JIRA.
If this patch is still in question, a conference call will be very helpful.
Let's schedule next week. [~ste...@apache.org] is traveling this week.
[~eddyxu] you have more comments since I revised the latest patch?
Thank you,
> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675430#comment-15675430
]
Hadoop QA commented on HADOOP-13114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
1 new + 160 unchanged - 0 fixed = 161 total (was 160) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
8s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839476/HADOOP-13114.05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c1e372d245b1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f05a9ce |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11094/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11094/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11094/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11094/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DistCp should have option to compress data on write
> ---
>
> Key: HADOOP-13114
> URL:
[jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675416#comment-15675416 ] Arun Suresh commented on HADOOP-11552: -- The patch looks good to me and the tests help understand the code path, Thanks [~sseth].. Minor nit: In Server::setDeferedResponse(), if setupResponse() throws an error, you still call sendDefferredResponse(). Wondering if you should just send a canned response.. or atleast something derived from the Exception. > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, > HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675414#comment-15675414 ] Brahma Reddy Battula commented on HADOOP-13742: --- [~kihwal] thanks a lot for your review and commit and thanks to [~vinayrpet] for additional review.. > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HADOOP-13114:
--
Attachment: HADOOP-13114.05.patch
Hi Suraj!
Thanks a lot for all your efforts to improve DistCp. My sincere apologies for
not paying attention to this issue. I'm sorry I was a bit busy when you had
asked and then never got back to this issue. Yongjun seems to want this in, so
we'll make another push for it.
Here's a rebase for the latest trunk. I'll try to review and test it in the
coming days.
> DistCp should have option to compress data on write
> ---
>
> Key: HADOOP-13114
> URL: https://issues.apache.org/jira/browse/HADOOP-13114
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Suraj Nayak
>Assignee: Suraj Nayak
>Priority: Minor
> Labels: distcp
> Attachments: HADOOP-13114-trunk_2016-05-07-1.patch,
> HADOOP-13114-trunk_2016-05-08-1.patch, HADOOP-13114-trunk_2016-05-10-1.patch,
> HADOOP-13114-trunk_2016-05-12-1.patch, HADOOP-13114.05.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> DistCp utility should have capability to store data in user specified
> compression format. This avoids one hop of compressing data after transfer.
> Backup strategies to different cluster also get benefit of saving one IO
> operation to and from HDFS, thus saving resources, time and effort.
> * Create an option -compressOutput defaulting to
> {{org.apache.hadoop.io.compress.BZip2Codec}}.
> * Users will be able to change codec with {{-D
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec}}
> * If distcp compression is enabled, suffix the filenames with default codec
> extension to indicate the file is compressed. Thus users can be aware of what
> codec was used to compress the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675307#comment-15675307
]
Lei (Eddy) Xu commented on HADOOP-13651:
This patch is in good shape now. Thanks for the good work, [~fabbri].
As we all agree that many other JIRAs depends on this patch, and the urgent of
merging it to trunk, I'd give a +1 for the {{v6}} patch.
> S3Guard: S3AFileSystem Integration with MetadataStore
> -
>
> Key: HADOOP-13651
> URL: https://issues.apache.org/jira/browse/HADOOP-13651
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13651-HADOOP-13345.001.patch,
> HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch,
> HADOOP-13651-HADOOP-13345.004.patch, HADOOP-13651-HADOOP-13345.005.patch,
> HADOOP-13651-HADOOP-13345.006.patch, v6-test-report.txt
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata
> consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8065) distcp should have an option to compress data while copying.
[
https://issues.apache.org/jira/browse/HADOOP-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675308#comment-15675308
]
Ravi Prakash commented on HADOOP-8065:
--
Hi Yongjun!
Thanks for rebasing the patch and your polishing touches. I think HADOOP-13114
might be the more appropriate JIRA for these changes (which Suraj kindly filed
at my request earlier.) since this patch does not compress *during* transfer;
only after transfer and before writing to HDFS.
- {{getCompressionCodcec}} has the same typo I pointed out to Suraj. He did
post updated patches on HADOOP-13114. I apologize for neglecting to review
those patches despite Suraj's requests.
- {{getCompressionCodcec}} also uses ReflectionUtils. I don't know if it'd be
better to use [this
pattern|https://github.com/apache/hadoop/blob/b4f1971ff1dd578353036d7a123fe83c27c1e803/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java#L159]
instead?
- We're still not using a CodecPool like I suggested earlier. The patch in
HADOOP-13114 actually is. Let me rebase and upload that. Could you please take
a look at that?
> distcp should have an option to compress data while copying.
>
>
> Key: HADOOP-8065
> URL: https://issues.apache.org/jira/browse/HADOOP-8065
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 0.20.2
>Reporter: Suresh Antony
>Assignee: Suraj Nayak
>Priority: Minor
> Labels: distcp
> Fix For: 0.20.2
>
> Attachments: HADOOP-8065-trunk_2015-11-03.patch,
> HADOOP-8065-trunk_2015-11-04.patch, HADOOP-8065-trunk_2016-04-29-4.patch,
> HADOOP-8065.005.patch, HADOOP-8065.006.patch, patch.distcp.2012-02-10
>
>
> We would like compress the data while transferring from our source system to
> target system. One way to do this is to write a map/reduce job to compress
> that after/before being transferred. This looks inefficient.
> Since distcp already reading writing data it would be better if it can
> accomplish while doing this.
> Flip side of this is that distcp -update option can not check file size
> before copying data. It can only check for the existence of file.
> So I propose if -compress option is given then file size is not checked.
> Also when we copy file appropriate extension needs to be added to file
> depending on compression type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
[ https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675281#comment-15675281 ] Aaron Fabbri commented on HADOOP-13793: --- This patch is applied on top of HADOOP-13651. (I will Submit Patch here when that one is merged.) > s3guard: add inconsistency injection, integration tests > --- > > Key: HADOOP-13793 > URL: https://issues.apache.org/jira/browse/HADOOP-13793 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-13793-HADOOP-13345.001.patch > > > Many of us share concerns that testing the consistency features of S3Guard > will be difficult if we depend on the rare and unpredictable occurrence of > actual inconsistency in S3 to exercise those code paths. > I think we should have a mechanism for injecting failure to force exercising > of the consistency codepaths in S3Guard. > Requirements: > - Integration tests that cause S3A to see the types of inconsistency we > address with S3Guard. > - These are deterministic integration tests. > Unit tests are possible as well, if we were to stub out the S3Client. That > may be less bang for the buck, though. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
[
https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13793:
--
Attachment: HADOOP-13793-HADOOP-13345.001.patch
Attaching v1 patch.
This adds a new integration test, {{ITestS3GuardListConsistency}} which sets up
the failure (inconsistency) injection and enables the in-memory MetadataStore,
then asserts that listStatus() is consistent even though visibility of one of
the paths it creates has been delayed.
To accomplish the inconsistency injection, I
- Promoted DefaultS3ClientFactory to a top-level class (it was an internal
class in S3ClientFactory interface)
- Pulled out the one-line call to construct a AmazonS3Client into an
overridable function.
- Create new class InconsistentS3ClientFactory which just subclases
DefaultS3ClientFactory and creates InconsistentS3Client instead of
AmazonS3Client.
- Created new class InconsistentS3Client which wraps AmazonS3Client, and delays
listObjects() visibility of any keys which match a given substring by 500 msec
after they are created.
I confirmed that, in ITestS3GuardListConsistency, if you comment out this line,
the test will fail:
{code}
+conf.setClass(S3Guard.S3_METADATA_STORE_IMPL, LocalMetadataStore.class,
+MetadataStore.class);
{code}
> s3guard: add inconsistency injection, integration tests
> ---
>
> Key: HADOOP-13793
> URL: https://issues.apache.org/jira/browse/HADOOP-13793
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13793-HADOOP-13345.001.patch
>
>
> Many of us share concerns that testing the consistency features of S3Guard
> will be difficult if we depend on the rare and unpredictable occurrence of
> actual inconsistency in S3 to exercise those code paths.
> I think we should have a mechanism for injecting failure to force exercising
> of the consistency codepaths in S3Guard.
> Requirements:
> - Integration tests that cause S3A to see the types of inconsistency we
> address with S3Guard.
> - These are deterministic integration tests.
> Unit tests are possible as well, if we were to stub out the S3Client. That
> may be less bang for the buck, though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[
https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675225#comment-15675225
]
Lei (Eddy) Xu commented on HADOOP-12928:
Hi, [~ozawa]
Sorry for late reply. I've tested ZKFC case, and updated both trunk and
branch-2 patches with Zookeeper 3.4.9.
One caveat was that, after [~ajisakaa] +1ed the trunk patch, I've already
committed that patch (w/o Zookeeper 3.4.9) into trunk, and then in the middle
of committing, received your comments. So this new trunk patch only updates
the zookeeper version, and branch-2 patch carries the actual changes in
{{branch-2}}.
Could you take another look?
> Update netty to 3.10.5.Final to sync with zookeeper
> ---
>
> Key: HADOOP-12928
> URL: https://issues.apache.org/jira/browse/HADOOP-12928
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.7.2
>Reporter: Hendy Irawan
>Assignee: Lei (Eddy) Xu
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12928-branch-2.00.patch,
> HADOOP-12928-branch-2.01.patch, HADOOP-12928-branch-2.02.patch,
> HADOOP-12928.01.patch, HADOOP-12928.02.patch, HADOOP-12928.03.patch,
> HDFS-12928.00.patch
>
>
> Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper
> 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927
> Pull request: https://github.com/apache/hadoop/pull/85
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-12928: --- Attachment: HADOOP-12928.03.patch > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928-branch-2.02.patch, > HADOOP-12928.01.patch, HADOOP-12928.02.patch, HADOOP-12928.03.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-12928: --- Attachment: HADOOP-12928-branch-2.02.patch > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928-branch-2.02.patch, > HADOOP-12928.01.patch, HADOOP-12928.02.patch, HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
[ https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13793 started by Aaron Fabbri. - > s3guard: add inconsistency injection, integration tests > --- > > Key: HADOOP-13793 > URL: https://issues.apache.org/jira/browse/HADOOP-13793 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > Many of us share concerns that testing the consistency features of S3Guard > will be difficult if we depend on the rare and unpredictable occurrence of > actual inconsistency in S3 to exercise those code paths. > I think we should have a mechanism for injecting failure to force exercising > of the consistency codepaths in S3Guard. > Requirements: > - Integration tests that cause S3A to see the types of inconsistency we > address with S3Guard. > - These are deterministic integration tests. > Unit tests are possible as well, if we were to stub out the S3Client. That > may be less bang for the buck, though. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
[ https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-13793: - Assignee: Aaron Fabbri > s3guard: add inconsistency injection, integration tests > --- > > Key: HADOOP-13793 > URL: https://issues.apache.org/jira/browse/HADOOP-13793 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > Many of us share concerns that testing the consistency features of S3Guard > will be difficult if we depend on the rare and unpredictable occurrence of > actual inconsistency in S3 to exercise those code paths. > I think we should have a mechanism for injecting failure to force exercising > of the consistency codepaths in S3Guard. > Requirements: > - Integration tests that cause S3A to see the types of inconsistency we > address with S3Guard. > - These are deterministic integration tests. > Unit tests are possible as well, if we were to stub out the S3Client. That > may be less bang for the buck, though. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11601) Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files
[
https://issues.apache.org/jira/browse/HADOOP-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675023#comment-15675023
]
Hudson commented on HADOOP-11601:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10855 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10855/])
HADOOP-11601. Enhance FS spec & tests to mandate (liuml07: rev
ae8849fe378e11b9db642ef7784c8e6a08199b96)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java
* (edit)
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
> Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty
> files
> ---
>
> Key: HADOOP-11601
> URL: https://issues.apache.org/jira/browse/HADOOP-11601
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, test
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-11601-001.patch, HADOOP-11601-002.patch,
> HADOOP-11601-003.patch, HADOOP-11601-004.patch, HADOOP-11601-005.patch,
> HADOOP-11601-006.patch, HADOOP-11601-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> HADOOP-11584 has shown that the contract tests are not validating that
> {{FileStatus.getBlocksize()}} must be >0 for any analytics jobs to partition
> workload correctly.
> Clarify in text and add test to do this. Test MUST be designed to work
> against eventually consistent filesystems where {{getFileStatus()}} may not
> be immediately visible, by retrying operation if FS declares it is an object
> store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11601) Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files
[
https://issues.apache.org/jira/browse/HADOOP-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-11601:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.8.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}} through {{branch-2.8}} branches. Thanks
[~ste...@apache.org] for your contribution; thanks [~templedf] and [~iwasakims]
for review.
> Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty
> files
> ---
>
> Key: HADOOP-11601
> URL: https://issues.apache.org/jira/browse/HADOOP-11601
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, test
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-11601-001.patch, HADOOP-11601-002.patch,
> HADOOP-11601-003.patch, HADOOP-11601-004.patch, HADOOP-11601-005.patch,
> HADOOP-11601-006.patch, HADOOP-11601-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> HADOOP-11584 has shown that the contract tests are not validating that
> {{FileStatus.getBlocksize()}} must be >0 for any analytics jobs to partition
> workload correctly.
> Clarify in text and add test to do this. Test MUST be designed to work
> against eventually consistent filesystems where {{getFileStatus()}} may not
> be immediately visible, by retrying operation if FS declares it is an object
> store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11601) Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files
[
https://issues.apache.org/jira/browse/HADOOP-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675000#comment-15675000
]
ASF GitHub Bot commented on HADOOP-11601:
-
Github user asfgit closed the pull request at:
https://github.com/apache/hadoop/pull/50
> Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty
> files
> ---
>
> Key: HADOOP-11601
> URL: https://issues.apache.org/jira/browse/HADOOP-11601
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, test
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-11601-001.patch, HADOOP-11601-002.patch,
> HADOOP-11601-003.patch, HADOOP-11601-004.patch, HADOOP-11601-005.patch,
> HADOOP-11601-006.patch, HADOOP-11601-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> HADOOP-11584 has shown that the contract tests are not validating that
> {{FileStatus.getBlocksize()}} must be >0 for any analytics jobs to partition
> workload correctly.
> Clarify in text and add test to do this. Test MUST be designed to work
> against eventually consistent filesystems where {{getFileStatus()}} may not
> be immediately visible, by retrying operation if FS declares it is an object
> store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674897#comment-15674897
]
Hadoop QA commented on HADOOP-11804:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client hadoop-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
27s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
13s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client
hadoop-client-modules/hadoop-client-api
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-invariants hadoop-dist
hadoop-client-modules . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 2s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
46s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}214m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674889#comment-15674889 ] Sean Busbey commented on HADOOP-11804: -- This shouldn't impact the hadoop-tools components AFAICT, since the shaded client jars are opt-in. Only one of the hadoop-tools components uses the hadoop-client dependency (which still works), the others directly reference more specific internal-facing dependencies. Do you have an example of the kind of problem you're expecting? > POC Hadoop Client w/o transitive dependencies > - > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch, > HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch, > HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch, > HADOOP-11804.9.patch > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12705) Upgrade Jackson 2.2.3 to 2.7.x or later
[ https://issues.apache.org/jira/browse/HADOOP-12705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674881#comment-15674881 ] Andrew Wang commented on HADOOP-12705: -- We're working on a shaded hadoop client at HADOOP-11804, after which classpath updates will have far more limited effect on clients. > Upgrade Jackson 2.2.3 to 2.7.x or later > --- > > Key: HADOOP-12705 > URL: https://issues.apache.org/jira/browse/HADOOP-12705 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > Attachments: HADOOP-12705.002.patch, HADOOP-12705.003.patch, > HADOOP-12705.01.patch, HADOOP-13050-001.patch > > > There's no rush to do this; this is just the JIRA to track versions. However, > without the upgrade, things written for Jackson 2.4.4 can break ( SPARK-12807) > being Jackson, this is a potentially dangerous update. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12953) New API for libhdfs to get FileSystem object as a proxy user
[
https://issues.apache.org/jira/browse/HADOOP-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674793#comment-15674793
]
Hadoop QA commented on HADOOP-12953:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 51s{color} | {color:orange} root: The patch generated 9 new + 127 unchanged
- 0 fixed = 136 total (was 127) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
1s{color} | {color:red} hadoop-common-project_hadoop-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 10s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-12953 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796835/HADOOP-12953.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
|
[jira] [Updated] (HADOOP-13279) Fix all Bad Practices
[
https://issues.apache.org/jira/browse/HADOOP-13279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated HADOOP-13279:
--
Summary: Fix all Bad Practices (was: Fix all Bad Practices flagged in
Fortify)
> Fix all Bad Practices
> -
>
> Key: HADOOP-13279
> URL: https://issues.apache.org/jira/browse/HADOOP-13279
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> The following code contain potential problems:
> {code}
> Unreleased Resource: Streams TopCLI.java:738
> Unreleased Resource: Streams Graph.java:189
> Unreleased Resource: Streams CgroupsLCEResourcesHandler.java:291
> Unreleased Resource: Streams UnmanagedAMLauncher.java:195
> Unreleased Resource: Streams CGroupsHandlerImpl.java:319
> Unreleased Resource: Streams TrafficController.java:629
> Portability Flaw: Locale Dependent Comparison TimelineWebServices.java:421
> Null Dereference ApplicationImpl.java:465
> Null Dereference VisualizeStateMachine.java:52
> Null Dereference ContainerImpl.java:1089
> Null Dereference QueueManager.java:219
> Null Dereference QueueManager.java:232
> Null Dereference ResourceLocalizationService.java:1016
> Null Dereference ResourceLocalizationService.java:1023
> Null Dereference ResourceLocalizationService.java:1040
> Null Dereference ResourceLocalizationService.java:1052
> Null Dereference ProcfsBasedProcessTree.java:802
> Null Dereference TimelineClientImpl.java:639
> Null Dereference LocalizedResource.java:206
> Code Correctness: Double-Checked Locking ResourceHandlerModule.java:142
> Code Correctness: Double-Checked Locking RMPolicyProvider.java:51
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674750#comment-15674750 ] Allen Wittenauer commented on HADOOP-11804: --- Do we know yet what the impact is going to be on the hadoop-tools components? Especially with the "on demand" jar detection at build time? > POC Hadoop Client w/o transitive dependencies > - > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch, > HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch, > HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch, > HADOOP-11804.9.patch > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674707#comment-15674707
]
Sean Busbey commented on HADOOP-11804:
--
Okay, I've updated [~andrew.wang]'s patch for avro master:
https://gist.github.com/busbey/caccd2fedda36409df4a319deb3e69c2
And given v9 locally installed, I get a clean run of {{./build.sh test}} in
{{lang/java}}.
the avro-mapred module ends up with this dep tree:
{code}
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-mapred ---
[INFO] org.apache.avro:avro-mapred:bundle:1.9.0-SNAPSHOT
[INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
[INFO] | +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- io.netty:netty:jar:3.5.13.Final:compile
[INFO] | +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | \- commons-lang:commons-lang:jar:2.4:compile
[INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
[INFO] +- org.apache.avro:avro-ipc:jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- org.apache.avro:avro:test-jar:tests:1.9.0-SNAPSHOT:test
[INFO] | +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
[INFO] | | +- com.google.guava:guava:jar:11.0.2:compile
[INFO] | | \-
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] | \- org.tukaani:xz:jar:1.5:compile
[INFO] +- org.easymock:easymock:jar:3.2:test
[INFO] | +- cglib:cglib-nodep:jar:2.2.2:test
[INFO] | \- org.objenesis:objenesis:jar:1.3:test
[INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] +- commons-codec:commons-codec:jar:1.9:compile
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
[INFO] +-
org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:provided
[INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] | \- commons-logging:commons-logging:jar:1.1.3:provided
[INFO] +- log4j:log4j:jar:1.2.17:provided
[INFO] +- junit:junit:jar:4.11:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime
{code}
Which looks about as I'd expect (maybe hadoop-client-api should be provided as
well, given the notes in the avro pom, but that's probably immaterial).
and avro-tools looks reasonable:
{code}
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-tools ---
[INFO] org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT
[INFO] +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
[INFO] | +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
[INFO] | | \- com.google.guava:guava:jar:11.0.2:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] | \- org.tukaani:xz:jar:1.5:compile
[INFO] +- org.apache.avro:avro:jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- org.apache.avro:avro-compiler:jar:1.9.0-SNAPSHOT:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | | \- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | \- joda-time:joda-time:jar:2.7:compile
[INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- io.netty:netty:jar:3.5.13.Final:compile
[INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
[INFO] +- org.apache.avro:avro-mapred:jar:1.9.0-SNAPSHOT:compile
[INFO] | \- commons-codec:commons-codec:jar:1.9:compile
[INFO] +- org.apache.avro:avro-mapred:test-jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- commons-cli:commons-cli:jar:1.2:compile
[INFO] +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] +- org.apache.avro:trevni-core:jar:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-avro:jar:1.9.0-SNAPSHOT:compile
[INFO] | \- org.apache.avro:avro-mapred:jar:hadoop2:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-core:jar:tests:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-avro:jar:tests:1.9.0-SNAPSHOT:compile
[INFO] +-
[jira] [Commented] (HADOOP-12953) New API for libhdfs to get FileSystem object as a proxy user
[ https://issues.apache.org/jira/browse/HADOOP-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674612#comment-15674612 ] Andres Perez commented on HADOOP-12953: --- This patch provides a good solution, given that it doesn't modify the signature of existing methods and just adds additional functionality. This is something that is still relevant still in 3.0.0-aplha > New API for libhdfs to get FileSystem object as a proxy user > > > Key: HADOOP-12953 > URL: https://issues.apache.org/jira/browse/HADOOP-12953 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.2 >Reporter: Uday Kale >Assignee: Uday Kale > Attachments: HADOOP-12953.001.patch, HADOOP-12953.002.patch > > > Secure impersonation in HDFS needs users to create proxy users and work with > those. In libhdfs, the hdfsBuilder accepts a userName but calls > FileSytem.get() or FileSystem.newInstance() with the user name to connect as. > But, both these interfaces use getBestUGI() to get the UGI for the given > user. This is not necessarily true for all services whose end-users would not > access HDFS directly, but go via the service to first get authenticated with > LDAP, then the service owner can impersonate the end-user to eventually > provide the underlying data. > For such services that authenticate end-users via LDAP, the end users are not > authenticated by Kerberos, so their authentication details wont be in the > Kerberos ticket cache. HADOOP_PROXY_USER is not a thread-safe way to get this > either. > Hence the need for the new API for libhdfs to get the FileSystem object as a > proxy user using the 'secure impersonation' recommendations. This approach is > secure since HDFS authenticates the service owner and then validates the > right for the service owner to impersonate the given user as allowed by > hadoop.proxyusers.* parameters of HDFS config. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674478#comment-15674478 ] Hudson commented on HADOOP-13742: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10854 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10854/]) HADOOP-13742. Expose NumOpenConnectionsPerUser as a metric. Brahma Reddy (kihwal: rev bd3735554fa5c3bc064c57ec78f4308430b14b48) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13742: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) I've committed it to trunk through branch-2.8. Thanks for working on this, Brahma. > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674412#comment-15674412 ] Kihwal Lee commented on HADOOP-13742: - +1 > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13825) KMSClientProvider is not using truststore and keystore provided in ssl-client.xml
[ https://issues.apache.org/jira/browse/HADOOP-13825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom James updated HADOOP-13825: --- Attachment: DelegationTokenAuthenticator.authenticate-crash Log file attached. Relevant lines: 10, 162 & 200 > KMSClientProvider is not using truststore and keystore provided in > ssl-client.xml > - > > Key: HADOOP-13825 > URL: https://issues.apache.org/jira/browse/HADOOP-13825 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Tom James > Attachments: DelegationTokenAuthenticator.authenticate-crash > > > When a KMSClientProvider is initialized it initializes a SSLFactory object > with its own trust store and key store (if > hadoop.ssl.require.client.cert=true). But during the course of execution, > KMSClientProvider.createConnection gets called, which in turn calls > KerberosDelegationTokenAuthenticator.authenticate(). Inside authenticate, > conn = (HttpURLConnection) url.openConnection() gets called which uses an > entirely different trust store (default trust store in JAVA_HOME; > "/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts" for me) and key > store (null). > This initializes the JVM trust store and keystore to the aforementioned trust > store and keystore. > This is causing problems down the line since all the specific certificates > (client, server, custom root certificates) are all in the trust store and > keystore specified in ssl-client. > Please find attached the log. > PS: > hadoop version > Hadoop 2.6.0-cdh5.7.1 > Subversion http://github.com/cloudera/hadoop -r > ae44a8970a3f0da58d82e0fc65275fff8deabffd > Compiled by jenkins on 2016-06-01T23:25Z > Compiled with protoc 2.5.0 > From source with checksum 298b68dc3b308983f04cb37e8416f13 > This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.7.1.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13814) A sample configuration of KMS HTTP Authentication signature will lead to misunderstanding
[
https://issues.apache.org/jira/browse/HADOOP-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674393#comment-15674393
]
Akira Ajisaka commented on HADOOP-13814:
Thanks [~masatana] for providing the patch. The change in the document
(index.md.vm) looks good to me, but the change in the setting (kms-site.xml) is
not good. The setting in kms-site.xml is used as default value, so changing the
setting is incompatible. Would you update the default value (kerberos) to
"none" instead of "sasl"? Updating "kerberos" to "none" does not change the
behavior, so this change is safe.
{code}
The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
{code}
In addition, I'm thinking we can improve the description to "The ZooKeeper
authentication type. Supported values are 'none' (default) and 'sasl'
(Kerberos)."
> A sample configuration of KMS HTTP Authentication signature will lead to
> misunderstanding
> -
>
> Key: HADOOP-13814
> URL: https://issues.apache.org/jira/browse/HADOOP-13814
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, documentation, kms
>Affects Versions: 3.0.0-alpha1
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-13814.patch
>
>
> In https://hadoop.apache.org/docs/current/hadoop-kms/index.html
> {code}
>
> hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type
> kerberos
>
> The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
>
>
> {code}
> It is very misleading. This configuration value should be 'sasl' when the
> kerberos authentication is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HADOOP-11804: - Attachment: HADOOP-11804.9.patch -9 - rebased to trunk (b2d4b7b) - fixed tabs now back to working on avro. > POC Hadoop Client w/o transitive dependencies > - > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch, > HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch, > HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch, > HADOOP-11804.9.patch > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13825) KMSClientProvider is not using truststore and keystore provided in ssl-client.xml
[ https://issues.apache.org/jira/browse/HADOOP-13825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674354#comment-15674354 ] Wei-Chiu Chuang commented on HADOOP-13825: -- Hi Tom thanks for reporting this, would you like to upload the log? > KMSClientProvider is not using truststore and keystore provided in > ssl-client.xml > - > > Key: HADOOP-13825 > URL: https://issues.apache.org/jira/browse/HADOOP-13825 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Tom James > > When a KMSClientProvider is initialized it initializes a SSLFactory object > with its own trust store and key store (if > hadoop.ssl.require.client.cert=true). But during the course of execution, > KMSClientProvider.createConnection gets called, which in turn calls > KerberosDelegationTokenAuthenticator.authenticate(). Inside authenticate, > conn = (HttpURLConnection) url.openConnection() gets called which uses an > entirely different trust store (default trust store in JAVA_HOME; > "/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts" for me) and key > store (null). > This initializes the JVM trust store and keystore to the aforementioned trust > store and keystore. > This is causing problems down the line since all the specific certificates > (client, server, custom root certificates) are all in the trust store and > keystore specified in ssl-client. > Please find attached the log. > PS: > hadoop version > Hadoop 2.6.0-cdh5.7.1 > Subversion http://github.com/cloudera/hadoop -r > ae44a8970a3f0da58d82e0fc65275fff8deabffd > Compiled by jenkins on 2016-06-01T23:25Z > Compiled with protoc 2.5.0 > From source with checksum 298b68dc3b308983f04cb37e8416f13 > This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.7.1.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13825) KMSClientProvider is not using truststore and keystore provided in ssl-client.xml
Tom James created HADOOP-13825: -- Summary: KMSClientProvider is not using truststore and keystore provided in ssl-client.xml Key: HADOOP-13825 URL: https://issues.apache.org/jira/browse/HADOOP-13825 Project: Hadoop Common Issue Type: Bug Components: kms Affects Versions: 2.6.0 Reporter: Tom James When a KMSClientProvider is initialized it initializes a SSLFactory object with its own trust store and key store (if hadoop.ssl.require.client.cert=true). But during the course of execution, KMSClientProvider.createConnection gets called, which in turn calls KerberosDelegationTokenAuthenticator.authenticate(). Inside authenticate, conn = (HttpURLConnection) url.openConnection() gets called which uses an entirely different trust store (default trust store in JAVA_HOME; "/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts" for me) and key store (null). This initializes the JVM trust store and keystore to the aforementioned trust store and keystore. This is causing problems down the line since all the specific certificates (client, server, custom root certificates) are all in the trust store and keystore specified in ssl-client. Please find attached the log. PS: hadoop version Hadoop 2.6.0-cdh5.7.1 Subversion http://github.com/cloudera/hadoop -r ae44a8970a3f0da58d82e0fc65275fff8deabffd Compiled by jenkins on 2016-06-01T23:25Z Compiled with protoc 2.5.0 >From source with checksum 298b68dc3b308983f04cb37e8416f13 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.7.1.jar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12705) Upgrade Jackson 2.2.3 to 2.7.x or later
[ https://issues.apache.org/jira/browse/HADOOP-12705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644932#comment-15644932 ] Sean Mackrory edited comment on HADOOP-12705 at 11/17/16 5:22 PM: -- Just attached one - didn't see any required changes other than the POM. One lingering concern I have is that while I understand the need for more coordination in branch-2, the minor changes required if we did move to Jackson 2.8.x in Hadoop 3.0 seem smaller to me than the danger in being locked into Jackson 2.7.x for the life of Hadoop 3.x, where incompatible changes soon become even worse. Granted, there's also no guarantee that Jackson 2.9.x won't *also* contain incompatible changes, so I could just be fighting a losing battle, but wanted to raise the concern. was (Author: mackrorysd): Just attached one - didn't see any required changes other than the POM. One lingering concern I have is that while I understand the need for more coordination in branch-2, the minor changes required if we did move to 2.8.x in 3.0 seem smaller to me than the danger in being locked into 2.7.x for the life of Hadoop 3, where incompatible changes soon become even worse. Granted, there's also no guarantee that 2.9.x won't *also* contain incompatible changes, so I could just be fighting a losing battle, but wanted to raise the concern. > Upgrade Jackson 2.2.3 to 2.7.x or later > --- > > Key: HADOOP-12705 > URL: https://issues.apache.org/jira/browse/HADOOP-12705 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > Attachments: HADOOP-12705.002.patch, HADOOP-12705.003.patch, > HADOOP-12705.01.patch, HADOOP-13050-001.patch > > > There's no rush to do this; this is just the JIRA to track versions. However, > without the upgrade, things written for Jackson 2.4.4 can break ( SPARK-12807) > being Jackson, this is a potentially dangerous update. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13820) [viewfs] Listfile gives complete Realm as User
[
https://issues.apache.org/jira/browse/HADOOP-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674284#comment-15674284
]
Hadoop QA commented on HADOOP-13820:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 257 unchanged - 0 fixed = 258 total (was 257) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13820 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839388/HADOOP-13820-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4f7f9ba42010 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b2d4b7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11091/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11091/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11091/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [viewfs] Listfile gives complete Realm as User
> --
>
> Key: HADOOP-13820
> URL: https://issues.apache.org/jira/browse/HADOOP-13820
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Archana T
>Assignee:
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[
https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674283#comment-15674283
]
Hadoop QA commented on HADOOP-13742:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 288 unchanged - 0 fixed = 290 total (was 288) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13742 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839386/HADOOP-13742-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4134611efa8f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b2d4b7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11090/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11090/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11090/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Expose "NumOpenConnectionsPerUser" as a metric
> --
>
> Key: HADOOP-13742
> URL: https://issues.apache.org/jira/browse/HADOOP-13742
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674275#comment-15674275
]
Sean Busbey commented on HADOOP-11804:
--
okay, {{mvn eclipse:eclipse}} makes them show up. I'm pretty sure we don't run
that as a part of the release processs, so these files shouldn't end up
published. we should add them to the ignore list for the rat-plugin, but that's
not related to this ticket specifically.
> POC Hadoop Client w/o transitive dependencies
> -
>
> Key: HADOOP-11804
> URL: https://issues.apache.org/jira/browse/HADOOP-11804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch,
> HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch,
> HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch
>
>
> make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to
> talk with a Hadoop cluster without seeing any of the implementation
> dependencies.
> see proposal on parent for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674238#comment-15674238
]
Sean Busbey commented on HADOOP-11804:
--
{code}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml
!?
/testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch
!? hadoop-client/.classpath
!? hadoop-client/.project
!? hadoop-client/.settings/org.eclipse.jdt.core.prefs
{code}
this looks like a gap between our building and yetus. Anyone have an idea of
how I get those temporary files to show up? I don't see any of them locally.
In any case, none of these are files from this patch (and the hadoop-client
directory shouldn't exist after the patch).
> POC Hadoop Client w/o transitive dependencies
> -
>
> Key: HADOOP-11804
> URL: https://issues.apache.org/jira/browse/HADOOP-11804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch,
> HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch,
> HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch
>
>
> make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to
> talk with a Hadoop cluster without seeing any of the implementation
> dependencies.
> see proposal on parent for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[
https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674203#comment-15674203
]
Hadoop QA commented on HADOOP-13742:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
13s{color} | {color:red} root in trunk failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} root in trunk failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
15s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in trunk failed with JDK v1.7.0_111.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
21s{color} | {color:red} root in the patch failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 21s{color}
| {color:red} root in the patch failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} root in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 10s{color}
| {color:red} root in the patch failed with JDK v1.7.0_111. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 288 unchanged - 0 fixed = 290 total (was 288) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_111.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 11s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-13742 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839386/HADOOP-13742-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 93875e462166 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b2d4b7b |
[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674194#comment-15674194
]
Rob Vesse commented on HADOOP-13824:
The command line is as follows:
{noformat}
hdfs dfs -D dfs.client.use.datanode.hostname=true -cat
hdfs://remote-namenode:8020/some/file
{noformat}
So I am attempting to talk to a Remote HDFS cluster where I can access the name
node but the data nodes are registered with internal IP addresses hence why I
want to use hostnames. I think the underlying cause is that the returned
hostnames are not resolvable on the machine where I run this command but since
the underlying error is suppressed I can't tell for certain.
Reproducing the specific case may be awkward because it appears to require
specific network configurations. However if you look at the code for
{{FsShell}} you should be able to see by inspection what I'm talking about.
Specifically there is a {{catch}} branch in {{FsShell.run()}} that does not log
the error and the {{displayError()}} method does not check that the provided
{{message}} argument is not {{null}}
I can probably provide a patch for this myself but I'm taking some time off
this weekend so it would be Monday at the earliest
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674087#comment-15674087 ] Brahma Reddy Battula edited comment on HADOOP-13742 at 11/17/16 4:20 PM: - bq.One minor thing in the latest patch is the new code in the test. server needs to be final, if you want this patch to go beyond trunk. Yes,,Thought of updating once after final review..Uploaded for same.. was (Author: brahmareddy): bq.One minor thing in the latest patch is the new code in the test. server needs to be final, if you want this patch to go beyond trunk. Yes,,Thought of updating once after final review.. Seems to be both are you ok with latest patch..? can I gohead with commit once after branc-2 patch jenkins run..? > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-13742: -- Attachment: HADOOP-13742-006.patch > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-006.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-13742: -- Attachment: (was: HADOOP-13742-branch-2.patch) > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674087#comment-15674087 ] Brahma Reddy Battula commented on HADOOP-13742: --- bq.One minor thing in the latest patch is the new code in the test. server needs to be final, if you want this patch to go beyond trunk. Yes,,Thought of updating once after final review.. Seems to be both are you ok with latest patch..? can I gohead with commit once after branc-2 patch jenkins run..? > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-branch-2.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-13742: -- Attachment: HADOOP-13742-branch-2.patch > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch, > HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742-branch-2.patch, > HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673990#comment-15673990
]
John Zhuge commented on HADOOP-13824:
-
Hi [~rvesse], thanks a lot for reporting the issue. It will make hadoop much
more usable and supportable to fix a issue like this. Could you please provide
the command line for your test?
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13824:
Assignee: John Zhuge
Labels: supportability (was: )
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[
https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673968#comment-15673968
]
Kihwal Lee commented on HADOOP-13742:
-
bq. That's a good idea and may be required to for detailed analysis about each
user's usage. In simple terms there could be separate instances of
RpcDetailedMetrics for each user, Just an Idea. I feel this could be done in a
follow-up jira and detailed discussions could be made.
I didn't mean to suggest per-user RpcDetailedMetrics. I meant per-user
connection metrics with cumulative counts. But the instantaneous numbers are
fine as well.
One minor thing in the latest patch is the new code in the test. {{server}}
needs to be {{final}}, if you want this patch to go beyond trunk.
> Expose "NumOpenConnectionsPerUser" as a metric
> --
>
> Key: HADOOP-13742
> URL: https://issues.apache.org/jira/browse/HADOOP-13742
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-13742-002.patch, HADOOP-13742-003.patch,
> HADOOP-13742-004.patch, HADOOP-13742-005.patch, HADOOP-13742.patch
>
>
> To track user level connections( How many connections for each user) in busy
> cluster where so many connections to server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13820) [viewfs] Listfile gives complete Realm as User
[
https://issues.apache.org/jira/browse/HADOOP-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673768#comment-15673768
]
Hadoop QA commented on HADOOP-13820:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 4 new + 257 unchanged - 0 fixed = 261 total (was 257) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13820 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839362/HADOOP-13820-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7565b5aff3f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aab9737 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11088/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11088/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11088/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [viewfs] Listfile gives complete Realm as User
> --
>
> Key: HADOOP-13820
> URL: https://issues.apache.org/jira/browse/HADOOP-13820
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Archana T
>Assignee:
[jira] [Commented] (HADOOP-13660) Upgrade commons-configuration version
[ https://issues.apache.org/jira/browse/HADOOP-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673682#comment-15673682 ] Wei-Chiu Chuang commented on HADOOP-13660: -- +1. The failed test is not reproducible in my local machine. > Upgrade commons-configuration version > - > > Key: HADOOP-13660 > URL: https://issues.apache.org/jira/browse/HADOOP-13660 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-13660-configuration2.001.patch, > HADOOP-13660.001.patch, HADOOP-13660.002.patch, HADOOP-13660.003.patch, > HADOOP-13660.004.patch, HADOOP-13660.005.patch, HADOOP-13660.006.patch, > HADOOP-13660.007.patch > > > We're currently pulling in version 1.6 - I think we should upgrade to the > latest 1.10. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13814) A sample configuration of KMS HTTP Authentication signature will lead to misunderstanding
[
https://issues.apache.org/jira/browse/HADOOP-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673651#comment-15673651
]
Hadoop QA commented on HADOOP-13814:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13814 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839345/HADOOP-13814.patch |
| Optional Tests | asflicense mvnsite unit xml |
| uname | Linux a0b241fffb81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aab9737 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11087/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11087/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> A sample configuration of KMS HTTP Authentication signature will lead to
> misunderstanding
> -
>
> Key: HADOOP-13814
> URL: https://issues.apache.org/jira/browse/HADOOP-13814
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, documentation, kms
>Affects Versions: 3.0.0-alpha1
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-13814.patch
>
>
> In https://hadoop.apache.org/docs/current/hadoop-kms/index.html
> {code}
>
> hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type
> kerberos
>
> The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
>
>
> {code}
> It is very misleading. This configuration value should be 'sasl' when the
> kerberos authentication is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13820) [viewfs] Listfile gives complete Realm as User
[
https://issues.apache.org/jira/browse/HADOOP-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673638#comment-15673638
]
Brahma Reddy Battula edited comment on HADOOP-13820 at 11/17/16 12:52 PM:
--
[~manojg] thanks for taking look..
Uploaded patch to address your comments and replaced remaining places also..
was (Author: brahmareddy):
[~manojg] thanks a taking look..
Uploaded patch to address your comments and replaced remaining places also..
> [viewfs] Listfile gives complete Realm as User
> --
>
> Key: HADOOP-13820
> URL: https://issues.apache.org/jira/browse/HADOOP-13820
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Archana T
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Attachments: HADOOP-13820-002.patch, HADOOP-13820.patch
>
>
> When defaultFS is configured as viewfs --
> fs.defaultFS
> viewfs://CLUSTER/
> List Files showing Realm as User --
> hdfs dfs -ls /
> Found 2 items
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir1
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13820) [viewfs] Listfile gives complete Realm as User
[
https://issues.apache.org/jira/browse/HADOOP-13820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-13820:
--
Attachment: HADOOP-13820-002.patch
[~manojg] thanks a taking look..
Uploaded patch to address your comments and replaced remaining places also..
> [viewfs] Listfile gives complete Realm as User
> --
>
> Key: HADOOP-13820
> URL: https://issues.apache.org/jira/browse/HADOOP-13820
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Reporter: Archana T
>Assignee: Brahma Reddy Battula
>Priority: Minor
> Attachments: HADOOP-13820-002.patch, HADOOP-13820.patch
>
>
> When defaultFS is configured as viewfs --
> fs.defaultFS
> viewfs://CLUSTER/
> List Files showing Realm as User --
> hdfs dfs -ls /
> Found 2 items
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir1
> -r-xr-xr-x - {color:red} h...@hadoop.com {color} hadoop 0
> 2016-11-07 15:31 /Dir2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13814) A sample configuration of KMS HTTP Authentication signature will lead to misunderstanding
[
https://issues.apache.org/jira/browse/HADOOP-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13814:
---
Status: Patch Available (was: Open)
> A sample configuration of KMS HTTP Authentication signature will lead to
> misunderstanding
> -
>
> Key: HADOOP-13814
> URL: https://issues.apache.org/jira/browse/HADOOP-13814
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, documentation, kms
>Affects Versions: 3.0.0-alpha1
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-13814.patch
>
>
> In https://hadoop.apache.org/docs/current/hadoop-kms/index.html
> {code}
>
> hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type
> kerberos
>
> The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
>
>
> {code}
> It is very misleading. This configuration value should be 'sasl' when the
> kerberos authentication is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11786) Fix Javadoc typos in org.apache.hadoop.fs.FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-11786: --- Fix Version/s: 3.0.0-alpha1 2.9.0 > Fix Javadoc typos in org.apache.hadoop.fs.FileSystem > > > Key: HADOOP-11786 > URL: https://issues.apache.org/jira/browse/HADOOP-11786 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 2.6.0 >Reporter: Chen He >Assignee: Andras Bokor >Priority: Trivial > Labels: newbie++ > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: HADOOP-11786.patch > > > /** > * Resets all statistics to 0. > * > * In order to reset, we add up all the thread-local statistics data, and > * set rootData to the negative of that. > * > * This may seem like a counterintuitive way to reset the statsitics. Why > * can't we just zero out all the thread-local data? Well, thread-local > * data can only be modified by the thread that owns it. If we tried to > * modify the thread-local data from this thread, our modification might > get > * interleaved with a read-modify-write operation done by the thread that > * owns the data. That would result in our update getting lost. > * > * The approach used here avoids this problem because it only ever reads > * (not writes) the thread-local data. Both reads and writes to rootData > * are done under the lock, so we're free to modify rootData from any > thread > * that holds the lock. > */ > etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673489#comment-15673489
]
Hadoop QA commented on HADOOP-11804:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client hadoop-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
11s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client
hadoop-client-modules/hadoop-client-api
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-invariants hadoop-dist
hadoop-client-modules . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 42s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
47s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA
[jira] [Updated] (HADOOP-13814) A sample configuration of KMS HTTP Authentication signature will lead to misunderstanding
[
https://issues.apache.org/jira/browse/HADOOP-13814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masahiro Tanaka updated HADOOP-13814:
-
Attachment: HADOOP-13814.patch
Thanks [~ajisakaa]! I wrote a patch for this.
> A sample configuration of KMS HTTP Authentication signature will lead to
> misunderstanding
> -
>
> Key: HADOOP-13814
> URL: https://issues.apache.org/jira/browse/HADOOP-13814
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, documentation, kms
>Affects Versions: 3.0.0-alpha1
>Reporter: Masahiro Tanaka
>Assignee: Masahiro Tanaka
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-13814.patch
>
>
> In https://hadoop.apache.org/docs/current/hadoop-kms/index.html
> {code}
>
> hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type
> kerberos
>
> The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
>
>
> {code}
> It is very misleading. This configuration value should be 'sasl' when the
> kerberos authentication is enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673439#comment-15673439
]
Rob Vesse commented on HADOOP-13824:
This isn't simply a case of me providing a bad argument at the command line. I
am using the generic {{-D}} to pass in a valid HDFS client option.
When I set that option to {{true}} I encounter this error, presumably because
the option leads to an {{IllegalArgumentException}} somewhere in the HDFS
client inner workings. But as I don't have a stack trace to go on I don't know
where to start looking.
When I set to {{false}} I avoid the NPE and the command runs (although still
errors elsewhere due to network configurations which setting the option to
{{true}} is a potential workaround for)
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Vesse updated HADOOP-13824:
---
Description:
The {{FsShell}} error handling assumes in {{displayError()}} that the
{{message}} argument is not {{null}}. However in the case where it is this
leads to a NPE which results in suppressing the actual error information since
a higher level of error handling kicks in and just dumps the stack trace of the
NPE instead.
e.g.
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
{noformat}
This is deeply unhelpful because depending on what the underlying error was
there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
{{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
which appears to be the underlying cause of my issue. Line 289 is where
{{displayError()}} is called for {{IllegalArgumentException}} handling and that
catch clause does not log the error.
was:
The {{FsShell}} error handling assumes in {{displayError()}} that the
{{message}} argument is not {{null}}. However in the case where it is this
leads to a NPE which results in suppressing the actual error information since
a higher level of error handling kicks in and just dumps the stack trace of the
NPE instead.
e.g.
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
{noformat}
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13824) FsShell can suppress the real error if no error message is present
Rob Vesse created HADOOP-13824:
--
Summary: FsShell can suppress the real error if no error message
is present
Key: HADOOP-13824
URL: https://issues.apache.org/jira/browse/HADOOP-13824
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.7.3, 2.7.1
Reporter: Rob Vesse
The {{FsShell}} error handling assumes in {{displayError()}} that the
{{message}} argument is not {{null}}. However in the case where it is this
leads to a NPE which results in suppressing the actual error information since
a higher level of error handling kicks in and just dumps the stack trace of the
NPE instead.
e.g.
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
[ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673327#comment-15673327 ] Steve Loughran commented on HADOOP-13651: - the MiniYarnCluster is HADOOP-13801; need to spend more time with git blame to idenfity what changed there. It's doing wordcount, and the format of the response has changed ... not sure it's in s3a tests, or the output format really has changed from text to hadoop writeable output. Root dir tests on object stores are always fussy. > S3Guard: S3AFileSystem Integration with MetadataStore > - > > Key: HADOOP-13651 > URL: https://issues.apache.org/jira/browse/HADOOP-13651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-13651-HADOOP-13345.001.patch, > HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch, > HADOOP-13651-HADOOP-13345.004.patch, HADOOP-13651-HADOOP-13345.005.patch, > HADOOP-13651-HADOOP-13345.006.patch, v6-test-report.txt > > > Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata > consistency and caching. > Implementation should have minimal overhead when no MetadataStore is > configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13823) s3a rename onto existing file must fail
Steve Loughran created HADOOP-13823:
---
Summary: s3a rename onto existing file must fail
Key: HADOOP-13823
URL: https://issues.apache.org/jira/browse/HADOOP-13823
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.7.3
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Critical
HIVE-15199 shows that s3a allows rename onto an existing file, which is
something HDFS, azure and s3n do not permit (though file:// does). This is
breaking bits of Hive, is an inconsistency with HDFS and a regression compared
to s3n semantics.
I propose: rejecting the rename on a file -> file rename if the destination
exists (easy) and changing the s3a.xml contract file to declare the behavior
change; this is needed for
{{AbstractContractRenameTest.testRenameFileOverExistingFile}} to handle the
changed semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673293#comment-15673293
]
Hadoop QA commented on HADOOP-13032:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
57s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
15s{color} | {color:green} root generated 0 new + 684 unchanged - 7 fixed = 684
total (was 691) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 58s{color} | {color:orange} root: The patch generated 37 new + 1569
unchanged - 41 fixed = 1606 total (was 1610) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
19s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
46s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-openstack in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
39s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
|
[jira] [Updated] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HADOOP-11804: - Attachment: HADOOP-11804.8.patch -08 - rebase to trunk (aab9737) - don't relocate the Hadoop config keys that start with "io." or "net." I'm going to work through the Avro example tomorrow morning, just want to get this put up for precommit and visibility in the mean time. I couldn't get an automated test easily. We essentially need an enforcer rule that can get the contents of String constants in a set of class files. I included a comment in the enforce-invariants pom of how I checked this manually. could use some better formatting. Not sure if it's worth pushing this check into dev-support and then invoking ant-run in the verify phase to run it or something like that. > POC Hadoop Client w/o transitive dependencies > - > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch, > HADOOP-11804.3.patch, HADOOP-11804.4.patch, HADOOP-11804.5.patch, > HADOOP-11804.6.patch, HADOOP-11804.7.patch, HADOOP-11804.8.patch > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
