[jira] [Commented] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627825#comment-15627825
]
Xiao Chen commented on HADOOP-13784:
The patch makes sense to me. Just verified it locally the previously-lingering
{{api}} dirs are gone. +1 let's get trunk fixed.
Thanks for the quick fix Andrew, and sorry for the miss in HADOOP-8500!
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[ https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13590: --- Attachment: (was: HADOOP-13590.09.patch) > Retry until TGT expires even if the UGI renewal thread encountered exception > > > Key: HADOOP-13590 > URL: https://issues.apache.org/jira/browse/HADOOP-13590 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0, 2.7.3, 2.6.4 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13590.01.patch, HADOOP-13590.02.patch, > HADOOP-13590.03.patch, HADOOP-13590.04.patch, HADOOP-13590.05.patch, > HADOOP-13590.06.patch, HADOOP-13590.07.patch, HADOOP-13590.08.patch, > HADOOP-13590.09.patch > > > The UGI has a background thread to renew the tgt. On exception, it > [terminates > itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014] > If something temporarily goes wrong that results in an IOE, even if it > recovered no renewal will be done and client will eventually fail to > authenticate. We should retry with our best effort, until tgt expires, in the > hope that the error recovers before that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[ https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13590: --- Attachment: HADOOP-13590.09.patch Seems my IDE was confused on indentation by the lambda. This should fix it. > Retry until TGT expires even if the UGI renewal thread encountered exception > > > Key: HADOOP-13590 > URL: https://issues.apache.org/jira/browse/HADOOP-13590 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.8.0, 2.7.3, 2.6.4 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13590.01.patch, HADOOP-13590.02.patch, > HADOOP-13590.03.patch, HADOOP-13590.04.patch, HADOOP-13590.05.patch, > HADOOP-13590.06.patch, HADOOP-13590.07.patch, HADOOP-13590.08.patch, > HADOOP-13590.09.patch > > > The UGI has a background thread to renew the tgt. On exception, it > [terminates > itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014] > If something temporarily goes wrong that results in an IOE, even if it > recovered no renewal will be done and client will eventually fail to > authenticate. We should retry with our best effort, until tgt expires, in the > hope that the error recovers before that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10300) Allowed deferred sending of call responses
[ https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-10300: --- Resolution: Fixed Fix Version/s: 2.7.4 Status: Resolved (was: Patch Available) I just committed the patch to branch-2.7. > Allowed deferred sending of call responses > -- > > Key: HADOOP-10300 > URL: https://issues.apache.org/jira/browse/HADOOP-10300 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Labels: BB2015-05-TBR > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: HADOOP-10300-branch-2.7.patch, HADOOP-10300.patch, > HADOOP-10300.patch, HADOOP-10300.patch > > > RPC handlers currently do not return until the RPC call completes and > response is sent, or a partially sent response has been queued for the > responder. It would be useful for a proxy method to notify the handler to > not yet the send the call's response. > An potential use case is a namespace handler in the NN might want to return > before the edit log is synced so it can service more requests and allow > increased batching of edits per sync. Background syncing could later trigger > the sending of the call response to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12718) Incorrect error message by fs -put local dir without permission
[
https://issues.apache.org/jira/browse/HADOOP-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627760#comment-15627760
]
Yongjun Zhang commented on HADOOP-12718:
Thanks [~cnauroth] and [~ste...@apache.org] for finding the potential
incompatibility of the earlier patch and the review/discussion, and thanks
[~jzhuge] for the continued work here.
I think patch rev 006 addressed the windows issue pointed out by Chris. One
minor comment, we should change the javadoc description to indicate that ACE
will be thrown when the given dir can't be read. I'm +1 after that's addressed.
And this will be marked as an incompatible change.
Hi Chris and Steve, do you have further comment?
Thanks.
> Incorrect error message by fs -put local dir without permission
> ---
>
> Key: HADOOP-12718
> URL: https://issues.apache.org/jira/browse/HADOOP-12718
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: John Zhuge
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HADOOP-12718.001.patch, HADOOP-12718.002.patch,
> HADOOP-12718.003.patch, HADOOP-12718.004.patch, HADOOP-12718.005.patch,
> HADOOP-12718.006.patch, TestFsShellCopyPermission-output.001.txt,
> TestFsShellCopyPermission-output.002.txt, TestFsShellCopyPermission.001.patch
>
>
> When the user doesn't have access permission to the local directory, the
> "hadoop fs -put" command prints a confusing error message "No such file or
> directory".
> {noformat}
> $ whoami
> systest
> $ cd /home/systest
> $ ls -ld .
> drwx--. 4 systest systest 4096 Jan 13 14:21 .
> $ mkdir d1
> $ sudo -u hdfs hadoop fs -put d1 /tmp
> put: `d1': No such file or directory
> {noformat}
> It will be more informative if the message is:
> {noformat}
> put: d1 (Permission denied)
> {noformat}
> If the source is a local file, the error message is ok:
> {noformat}
> put: f1 (Permission denied)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[
https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627756#comment-15627756
]
Hadoop QA commented on HADOOP-13590:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 6 new + 145 unchanged - 0 fixed = 151 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
33s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Redundant nullcheck of tgt, which is known to be non-null in
org.apache.hadoop.security.UserGroupInformation$1.run() Redundant null check
at UserGroupInformation.java:is known to be non-null in
org.apache.hadoop.security.UserGroupInformation$1.run() Redundant null check
at UserGroupInformation.java:[line 1011] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836501/HADOOP-13590.09.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3c650cd37659 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fc2b69e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10959/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10959/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10959/testReport/ |
| modules | C:
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627709#comment-15627709
]
Xiao Chen commented on HADOOP-12453:
Thanks [~xyao] for the prompt response. Some more comments:
- Jenkins -1'ed because the new KMSDelegationToken file was partially patched
in 01. I had to apply 00 first, then 01 to get a clean look.
- Unused import {{KMSClientProvider}} in KMSAuthenticationFilter.java
- The new KMSDTI is still defined in hdfs?
- Don't think we need to keep {{TOKEN_KIND}} and {{TOKEN_KIND_STR}} in KMSCP.
Let's just use KMSDTI. It seems currently they're only used in TestKMS, which
we could also clean up with a static import of KMSDTI.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch, HADOOP-12453.01.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[
https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13590:
---
Attachment: HADOOP-13590.09.patch
Patch 9 to address the comments. I find the lambda {{wait()}} easier to use
than {{GenericTestUtil#waitFor}}. :) Thank you Steve.
Created HADOOP-13785 for the clean up.
> Retry until TGT expires even if the UGI renewal thread encountered exception
>
>
> Key: HADOOP-13590
> URL: https://issues.apache.org/jira/browse/HADOOP-13590
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.4
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13590.01.patch, HADOOP-13590.02.patch,
> HADOOP-13590.03.patch, HADOOP-13590.04.patch, HADOOP-13590.05.patch,
> HADOOP-13590.06.patch, HADOOP-13590.07.patch, HADOOP-13590.08.patch,
> HADOOP-13590.09.patch
>
>
> The UGI has a background thread to renew the tgt. On exception, it
> [terminates
> itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014]
> If something temporarily goes wrong that results in an IOE, even if it
> recovered no renewal will be done and client will eventually fail to
> authenticate. We should retry with our best effort, until tgt expires, in the
> hope that the error recovers before that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13785) Clean up tests that conditionally handle IBM_JAVA
Xiao Chen created HADOOP-13785:
--
Summary: Clean up tests that conditionally handle IBM_JAVA
Key: HADOOP-13785
URL: https://issues.apache.org/jira/browse/HADOOP-13785
Project: Hadoop Common
Issue Type: Improvement
Components: test
Affects Versions: 2.8.0
Reporter: Xiao Chen
Priority: Minor
Initially from a [review
comment|https://issues.apache.org/jira/browse/HADOOP-13590?focusedCommentId=15625141=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15625141]
by [~ste...@apache.org].
In some tests we see code like below:
{code}
if (PlatformName.IBM_JAVA) {
options.put("useKeytab", keytab.getPath());
options.put("credsType", "both");
} else {
options.put("keyTab", keytab.getPath());
options.put("useKeyTab", "true");
options.put("storeKey", "true");
options.put("doNotPrompt", "true");
options.put("useTicketCache", "true");
options.put("renewTGT", "true");
options.put("isInitiator", Boolean.toString(true));
}
{code}
We should extract a util function to some degree, and clean them up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[
https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627575#comment-15627575
]
Hadoop QA commented on HADOOP-12928:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
33s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
33s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 30s{color} | {color:orange} root: The patch generated 1 new + 89 unchanged -
1 fixed = 90 total (was 90) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-project in the patch passed with JDK
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m
13s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_111.
{color} |
| {color:red}-1{color} | {color:red} unit {color} |
[jira] [Work started] (HADOOP-13767) Aliyun Connection broken when idle then 1 minutes or build than 3 hours
[ https://issues.apache.org/jira/browse/HADOOP-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13767 started by Genmao Yu. -- > Aliyun Connection broken when idle then 1 minutes or build than 3 hours > --- > > Key: HADOOP-13767 > URL: https://issues.apache.org/jira/browse/HADOOP-13767 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627435#comment-15627435 ] Genmao Yu commented on HADOOP-13769: [~ste...@apache.org] Take a look please > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. > -- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13768) AliyunOSS: DeleteObjectsRequest has 1000 objects limit
[ https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627430#comment-15627430 ] Genmao Yu commented on HADOOP-13768: [~drankye] Take a look please. > AliyunOSS: DeleteObjectsRequest has 1000 objects limit > --- > > Key: HADOOP-13768 > URL: https://issues.apache.org/jira/browse/HADOOP-13768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627423#comment-15627423
]
Hadoop QA commented on HADOOP-12453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-12453 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-12453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836491/HADOOP-12453.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10958/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch, HADOOP-12453.01.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-12453:
Attachment: HADOOP-12453.01.patch
Thanks [~xiaochen] for the heads up. I've updated the patch to address the
comments. Decide to keep the KMSDelegationToken class as a placeholder for the
kms-dt related constants and the identifier class. Please review, thanks!
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch, HADOOP-12453.01.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627416#comment-15627416 ] Fei Hui commented on HADOOP-13773: -- Thanks. I'll take care in the future. > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Fix For: 2.8.0 > > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13738) DiskChecker should perform some disk IO
[ https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627291#comment-15627291 ] Hudson commented on HADOOP-13738: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10750/]) HADOOP-13738. DiskChecker should perform some disk IO. (arp: rev 1b6ecaf016aaf7f6a09a4d576294b5e0a6850a1f) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestDiskChecker.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java > DiskChecker should perform some disk IO > --- > > Key: HADOOP-13738 > URL: https://issues.apache.org/jira/browse/HADOOP-13738 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch, > HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch > > > DiskChecker can fail to detect total disk/controller failures indefinitely. > We have seen this in real clusters. DiskChecker performs simple > permissions-based checks on directories which do not guarantee that any disk > IO will be attempted. > A simple improvement is to write some data and flush it to the disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13738) DiskChecker should perform some disk IO
[
https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-13738:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Target Version/s: (was: 2.9.0)
Status: Resolved (was: Patch Available)
Thank you all for the reviews and discussion.
Pushed for 2.9.0. The branch-2 commit required the following delta wrt trunk to
compile:
{code}
private static AtomicReference fileIoProvider =
- new AtomicReference<>(new DefaultFileIoProvider());
+ new AtomicReference(new DefaultFileIoProvider());
{code}
> DiskChecker should perform some disk IO
> ---
>
> Key: HADOOP-13738
> URL: https://issues.apache.org/jira/browse/HADOOP-13738
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch,
> HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch
>
>
> DiskChecker can fail to detect total disk/controller failures indefinitely.
> We have seen this in real clusters. DiskChecker performs simple
> permissions-based checks on directories which do not guarantee that any disk
> IO will be attempted.
> A simple improvement is to write some data and flush it to the disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627262#comment-15627262
]
Hadoop QA commented on HADOOP-13781:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836479/HADOOP-13781.4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cf853e8b4183 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bd7f591 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10955/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10955/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>
[jira] [Commented] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627260#comment-15627260
]
Hadoop QA commented on HADOOP-13784:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
6s{color} | {color:green} hadoop-project-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13784 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836483/HADOOP-13784.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 4a7601b2a538 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bd7f591 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10957/testReport/ |
| modules | C: hadoop-project-dist U: hadoop-project-dist |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10957/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627251#comment-15627251
]
Hadoop QA commented on HADOOP-13427:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 43s{color} | {color:orange} root: The patch generated 8 new + 850 unchanged
- 10 fixed = 858 total (was 860) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} hadoop-tools/hadoop-rumen generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 40s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 38s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
25s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 23s{color}
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-server-sharedcachemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 13s{color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
43s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
|
[jira] [Comment Edited] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627182#comment-15627182
]
Andres Perez edited comment on HADOOP-13781 at 11/2/16 1:04 AM:
Waiting before throwing the exeception
{{ActiceStanbyElector#reEstablishSession}} is just a waste of time, inverting
the order on this patch
was (Author: aaperezl):
Waiting before throwing the exeception
{{ActiceStanbyElector#reEstablishSession}} is just a waste of time, reverting
the order on this patch
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13784:
-
Status: Patch Available (was: Open)
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13784:
-
Attachment: HADOOP-13784.001.patch
Patch attached. We now point the plugin so it outputs within the target
directory.
I ran the site build and browsed around, the Javadocs are still available at
"/api".
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8500) Fix javadoc jars to not contain entire target directory
[ https://issues.apache.org/jira/browse/HADOOP-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627231#comment-15627231 ] Andrew Wang commented on HADOOP-8500: - Thanks for the catch Allen, filed HADOOP-13784 with a fix. > Fix javadoc jars to not contain entire target directory > --- > > Key: HADOOP-8500 > URL: https://issues.apache.org/jira/browse/HADOOP-8500 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.0.0-alpha > Environment: N/A >Reporter: EJ Ciramella >Assignee: Andrew Wang >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-8500.001.patch, HADOOP-8500.002.patch, > HADOOP-8500.003.patch, HADOOP-8500.patch, site-redo.tar > > Original Estimate: 24h > Remaining Estimate: 24h > > The javadoc jars contain the contents of the target directory - which > includes classes and all sorts of binary files that it shouldn't. > Sometimes the resulting javadoc jar is 10X bigger than it should be. > The fix is to reconfigure maven to use "api" as it's destDir for javadoc > generation. > I have a patch/diff incoming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13784) Output javadoc inside the target directory
Andrew Wang created HADOOP-13784:
Summary: Output javadoc inside the target directory
Key: HADOOP-13784
URL: https://issues.apache.org/jira/browse/HADOOP-13784
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Andrew Wang
Assignee: Andrew Wang
HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
javadoc dirs being created outside target folders:
{noformat}
hadoop-common-project/hadoop-common/api/
hadoop-hdfs-project/hadoop-hdfs-client/api/
hadoop-hdfs-project/hadoop-hdfs/api/
{noformat}
Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.4.patch
Waiting before throwing the exeception
{{ActiceStanbyElector#reEstablishSession}} is just a waste of time, reverting
the order on this patch
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-12928: --- Attachment: HADOOP-12928-branch-2.01.patch Hi, [~ozawa], Zookeeper 3.4.9 and 3.5.0 have been released. Would you take a look of branch-2 patch to see whether it is ok to commit? Thanks! > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627137#comment-15627137
]
Hadoop QA commented on HADOOP-13781:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
54s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836465/HADOOP-13781.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0ce954091549 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aacf214 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10954/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10954/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>
[jira] [Commented] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
[ https://issues.apache.org/jira/browse/HADOOP-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627122#comment-15627122 ] Hudson commented on HADOOP-13583: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10749 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10749/]) HADOOP-13583. Incorporate checkcompatibility script which runs Java API (wang: rev 0cd22d66691fdbc3fb3e0a35ad1625236c3ae3f7) * (edit) BUILDING.txt * (add) dev-support/bin/checkcompatibility.py > Incorporate checkcompatibility script which runs Java API Compliance Checker > > > Key: HADOOP-13583 > URL: https://issues.apache.org/jira/browse/HADOOP-13583 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13583.001.patch, HADOOP-13583.002.patch, > HADOOP-13583.003.patch > > > Based on discussion at YETUS-445, this code can't go there, but it's still > very useful for release managers. A similar variant of this script has been > used for a while by Apache HBase and Apache Kudu, and IMO JACC output is > easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627123#comment-15627123
]
Hudson commented on HADOOP-13717:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10749 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10749/])
HADOOP-13717. Normalize daemonization behavior of the diskbalancer with (wang:
rev bd7f5911c726c505a654d5d30b7dc5a92dca05bd)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13667) Fix typing mistake of inline document in hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627109#comment-15627109
]
Andrew Wang commented on HADOOP-13667:
--
I don't know much about Ganglia either, but since this is just additional
documentation, I think it's okay for commit.
> Fix typing mistake of inline document in hadoop-metrics2.properties
> ---
>
> Key: HADOOP-13667
> URL: https://issues.apache.org/jira/browse/HADOOP-13667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Rui Gao
>Assignee: Rui Gao
> Attachments: HADOOP-13667.2.patch, HADOOP-13667.patch
>
>
> Fix typing mistake of inline document in hadoop-metrics2.properties.
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcesName{code} should be
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcessName{code}
> And also could add examples into the inline document for easier understanding
> of metrics tag related configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13717:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Thanks Allen for discussion and review, I've committed this to trunk.
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
[ https://issues.apache.org/jira/browse/HADOOP-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13583: - Resolution: Fixed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Thanks for the reviews Robert! Committed to trunk, branch-2, branch-2.8. > Incorporate checkcompatibility script which runs Java API Compliance Checker > > > Key: HADOOP-13583 > URL: https://issues.apache.org/jira/browse/HADOOP-13583 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13583.001.patch, HADOOP-13583.002.patch, > HADOOP-13583.003.patch > > > Based on discussion at YETUS-445, this code can't go there, but it's still > very useful for release managers. A similar variant of this script has been > used for a while by Apache HBase and Apache Kudu, and IMO JACC output is > easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627043#comment-15627043 ] NITIN VERMA commented on HADOOP-13783: -- Thanks [~liuml07] > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627034#comment-15627034 ] Mingliang Liu commented on HADOOP-13783: Hi [~nitin.ve...@gmail.com], I added you as Hadoop Contributor, and assigned this JIRA to you. Looking forward to your contribution! > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.3.patch
Added patch with checkstyle fix
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13783: --- Assignee: NITIN VERMA (was: Mingliang Liu) > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned HADOOP-13783: -- Assignee: Mingliang Liu > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: azure >Reporter: NITIN VERMA >Assignee: Mingliang Liu > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627012#comment-15627012
]
Hadoop QA commented on HADOOP-13781:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 49 unchanged - 0 fixed = 50 total (was 49) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
10s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836453/HADOOP-13781.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a4bde0d3c9e6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fa1512a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10952/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10952/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10952/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key:
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627005#comment-15627005
]
Hadoop QA commented on HADOOP-12453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-12453 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-12453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774399/HADOOP-12453.00.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10953/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626987#comment-15626987
]
Xiao Chen commented on HADOOP-12453:
Hi [~xyao],
Thanks for reporting this and providing a patch. Looks great overall!
A couple of comments/questions:
- It's been a while so needs a rebase. Can remove KMSCP's new
{{TOKEN_KIND_STR}} too.
- Should this really go to hdfs's services, as opposed to common?
- Can this work with just the KMSDelegationTokenIdentifier class (without the
outer KMSDelegationToken class)?
If you're busy, I'd be happy to continue to work on this.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626985#comment-15626985 ] NITIN VERMA commented on HADOOP-13783: -- I don't have permission to assign this JIRA. Could someone assign this to me? > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Bug > Components: azure >Reporter: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626982#comment-15626982
]
Mingliang Liu commented on HADOOP-13650:
Thanks for the nice work, [~eddyxu]!
{quote}
One question I have in the implementation is that, for initialize() / destroy
functions, can we provide a version of such functions that do not take
S3FIleSystem as parameters (i.e., taking Configuration instead)? Providing s3a
path in s3a s3guard init -m s3a://bucket/path seems be not necessary.
{quote}
Basically we can create a S3AFileSystem via URI (bucket) and conf; and I
thought we could initialize the s3a fs object anywhere.
The DDBMetadataStore needs the region, bucket (default table name), AWS keys
etc, among which the bucket is not specified in the configuration. Newly added
config key DDB_TABLE_NAME_KEY can be helpful in this case. In short, I think
it's possible to achieve this. Pssing s3a URI along with the MetadataStore URI
seems not necessary to me either in the CLI tool. You can assume I can
construct a DDBMetadataStore using configuration simply. I'll let you know if I
can't. This will be considered in the [HADOOP-13449] v2 patch, on its way.
I'll come back to review this patch this week.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13736) Change PathMetadata to hold S3AFileStatus instead of FileStatus.
[
https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626981#comment-15626981
]
Mingliang Liu commented on HADOOP-13736:
I'm holding on commit. Thanks,
> Change PathMetadata to hold S3AFileStatus instead of FileStatus.
>
>
> Key: HADOOP-13736
> URL: https://issues.apache.org/jira/browse/HADOOP-13736
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13736-HADOOP-13345.000.patch,
> HADOOP-13736-HADOOP-13345.001.patch, HADOOP-13736.000.patch,
> HADOOP-13736.wip-01.patch
>
>
> {{S3AFileStatus}} is implemented differently with {{FileStatus}}, for
> instance {{S3AFileStatus#isEmptyDirectory()}} is not implemented in
> {{FileStatus()}}. And {{access_time}}, {{block_replication}}, {{owner}},
> {{group}} and a few other fields are not meaningful in {{S3AFileStatus}}.
> So in the scope of {{S3guard}}, it should use {{S3AFileStatus}} in instead
> of {{FileStatus}} in {{PathMetadaa}} to avoid casting the types back and
> forth in S3A.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13783) Improve efficiency of WASB over page blobs
NITIN VERMA created HADOOP-13783: Summary: Improve efficiency of WASB over page blobs Key: HADOOP-13783 URL: https://issues.apache.org/jira/browse/HADOOP-13783 Project: Hadoop Common Issue Type: Bug Components: azure Reporter: NITIN VERMA 1) Add telemetry to WASB driver. WASB driver is lack of any log or telemetry which makes trouble shoot very difficult. For example, we don’t know where is high e2e latency between HBase and Azure storage came from when Azure storage server latency was very low. Also we don’t know why WASB can only do 166 IOPs which is way below azure storage 500 IOPs. And we had several incidents before related to storage latency, because of lacking logs, we couldn’t find the ownership of the incident quickly. 2) Resolving the hot spotting issue when WASB driver partition the azure page blobs by changing the key. Current key design is causing the hot spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
[ https://issues.apache.org/jira/browse/HADOOP-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626926#comment-15626926 ] Robert Kanter commented on HADOOP-13583: Sounds good to me. > Incorporate checkcompatibility script which runs Java API Compliance Checker > > > Key: HADOOP-13583 > URL: https://issues.apache.org/jira/browse/HADOOP-13583 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-13583.001.patch, HADOOP-13583.002.patch, > HADOOP-13583.003.patch > > > Based on discussion at YETUS-445, this code can't go there, but it's still > very useful for release managers. A similar variant of this script has been > used for a while by Apache HBase and Apache Kudu, and IMO JACC output is > easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13673) Update sbin/start-* and sbin/stop-* to be smarter
[
https://issues.apache.org/jira/browse/HADOOP-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626912#comment-15626912
]
Hadoop QA commented on HADOOP-13673:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 new + 75 unchanged - 0 fixed =
77 total (was 75) {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}
0m 10s{color} | {color:orange} The patch generated 8 new + 122 unchanged - 2
fixed = 130 total (was 124) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
52s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 7s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13673 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836422/HADOOP-13673.00.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall xml |
| uname | Linux 02e23a536e4f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk /
[jira] [Updated] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-13650:
---
Attachment: HADOOP-13650-HADOOP-13345.000.patch
Uploaded a working-in-progress patch. The patch requires that the
{{MetadataStore}} interface and {{DynamoDB}} metadata store to be ready.
[~liuml07], One question I have in the implementation is that, for
{{initialize()}} / {{destroy}} functions, can we provide a version of such
functions that do not take {{S3FIleSystem}} as parameters (i.e., taking
{{Configuration}} instead)? Providing s3a path in {{s3a s3guard init -m
s3a://bucket/path}} seems be not necessary.
[~ste...@apache.org] I've added a new {{s3a}} command in this patch.
Additionally, I proposed to use {{URI}} as metadata store table name, so that
we can support {{dynamodb://}}, {{mysql://}}, {{local://}} and etc.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.2.patch
The {{ActiveStandbyElector#reEstablishSession}} method does not throws
{{IOException}} nor {{KeeperException}} making the Zookeeper connection fail
silently when invoking {{ZKFailoverController#initZK}} making the Unit test
fail when the {{failFast}} paremeter is set to {{false}}
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626888#comment-15626888 ] Ravi Prakash commented on HADOOP-13773: --- Awesome! Sounds good Andrew! Thanks. I'll take care in the future. > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Fix For: 2.8.0 > > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13738) DiskChecker should perform some disk IO
[ https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626847#comment-15626847 ] Kihwal Lee commented on HADOOP-13738: - +1 > DiskChecker should perform some disk IO > --- > > Key: HADOOP-13738 > URL: https://issues.apache.org/jira/browse/HADOOP-13738 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch, > HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch > > > DiskChecker can fail to detect total disk/controller failures indefinitely. > We have seen this in real clusters. DiskChecker performs simple > permissions-based checks on directories which do not guarantee that any disk > IO will be attempted. > A simple improvement is to write some data and flush it to the disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13738) DiskChecker should perform some disk IO
[ https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626837#comment-15626837 ] Arpit Agarwal commented on HADOOP-13738: Thanks for the review [~xyao]. [~kihwal], it didn't sound like you had any objections to the proposed approach but I'll wait until the end of this week before committing. Filed HDFS-11086 to separately address some more improvements in DN's use of DiskChecker. This will address the check taking minutes failure case you brought up. > DiskChecker should perform some disk IO > --- > > Key: HADOOP-13738 > URL: https://issues.apache.org/jira/browse/HADOOP-13738 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch, > HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch > > > DiskChecker can fail to detect total disk/controller failures indefinitely. > We have seen this in real clusters. DiskChecker performs simple > permissions-based checks on directories which do not guarantee that any disk > IO will be attempted. > A simple improvement is to write some data and flush it to the disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626832#comment-15626832 ] Andrew Wang commented on HADOOP-13773: -- Hi Ravi, precommit is multi-branch now, if you follow the naming convention of e.g. HADOOP-13733.branch-2.001.patch. See: https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch The precommit bot will also detect the branch correctly for github PRs, though if there's both a PR and a patch (as on this JIRA), the precommit bot will always use the PR. When this has happened in the past, I've resorted to filing a new JIRA to unstick precommit. > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Fix For: 2.8.0 > > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
[ https://issues.apache.org/jira/browse/HADOOP-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626818#comment-15626818 ] Andrew Wang commented on HADOOP-13583: -- Thanks for reviewing Robert. The two tabs are example output in a comment for some parsing code, so I'd prefer not to change it. I can chmod +x the file before commit. Robert, this good with you? > Incorporate checkcompatibility script which runs Java API Compliance Checker > > > Key: HADOOP-13583 > URL: https://issues.apache.org/jira/browse/HADOOP-13583 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-13583.001.patch, HADOOP-13583.002.patch, > HADOOP-13583.003.patch > > > Based on discussion at YETUS-445, this code can't go there, but it's still > very useful for release managers. A similar variant of this script has been > used for a while by Apache HBase and Apache Kudu, and IMO JACC output is > easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626805#comment-15626805 ] Ravi Prakash commented on HADOOP-13773: --- Andrew! The precommit was known to fail because the patch applied only to branch-2 (not trunk) > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Fix For: 2.8.0 > > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626772#comment-15626772
]
Hadoop QA commented on HADOOP-13781:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836426/HADOOP-13781.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8541d46fa4f8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 76893a4 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10950/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10950/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10950/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL:
[jira] [Updated] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13427:
---
Summary: Eliminate needless uses of FileSystem#{exists(), isFile(),
isDirectory()} (was: Eliminate needless uses of FileSystem.exists, isFile,
isDirectory )
> Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
> -
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch,
> HADOOP-13427.003.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13427) Eliminate needless uses of FileSystem.exists, isFile, isDirectory
[ https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13427: --- Priority: Major (was: Minor) > Eliminate needless uses of FileSystem.exists, isFile, isDirectory > -- > > Key: HADOOP-13427 > URL: https://issues.apache.org/jira/browse/HADOOP-13427 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch, > HADOOP-13427.003.patch > > > We're cleaning up Hive and Spark's use of FileSystem.exists, because it is > often the case we see code of exists+open, exists+delete, when the exists > probe is needless. Against object stores, expensive needless. > Hadoop can set an example here by stripping them out. It will also show where > there are opportunities to optimise things better and/or improve reporting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13427) Eliminate needless uses of FileSystem.exists, isFile, isDirectory
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13427:
---
Attachment: HADOOP-13427.003.patch
+1 the v2 patch.
I rebased the patch from {{trunk}} branch as the v2 patch can not apply
cleanly. It may be interesting if [~ste...@apache.org] can +1 for your own
patch, see [#HADOOP-13427.003.patch].
I don't quite get the following code snippet:
{code:title=SwiftTestUtils.java}
496 throw (IOException)new FileNotFoundException(message + ": not
found "
497 + path + " in " + path.getParent() + ": " + e + " -- "
498+ ls(fileSystem, path.getParent())).initCause(e);
{code}
and
{code:title=FileSystemApplicationHistoryStore.java}
658 try {
659 fs.getFileStatus(applicationHistoryFile);
660 } catch (FileNotFoundException e) {
661 throw (FileNotFoundException) new FileNotFoundException
662 ("History file for application " + appId + " is not found: "
+ e)
663 .initCause(e);
{code}
> Eliminate needless uses of FileSystem.exists, isFile, isDirectory
> --
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch,
> HADOOP-13427.003.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13782) Make MutableRates metrics thread-local write, aggregate-on-read
Erik Krogen created HADOOP-13782:
Summary: Make MutableRates metrics thread-local write,
aggregate-on-read
Key: HADOOP-13782
URL: https://issues.apache.org/jira/browse/HADOOP-13782
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Reporter: Erik Krogen
Assignee: Erik Krogen
Currently the {{MutableRates}} metrics class serializes all writes to metrics
it contains because of its use of {{MetricsRegistry.add()}} (i.e., even two
increments of unrelated metrics contained within the same {{MutableRates}}
object will serialize w.r.t. each other). This class is used by
{{RpcDetailedMetrics}}, which may have many hundreds of threads contending to
modify these metrics. Instead we should allow updates to unrelated metrics
objects to happen concurrently. To do so we can let each thread locally collect
metrics, and on a {{snapshot}}, aggregate the metrics from all of the threads.
I have collected some benchmark performance numbers in HADOOP-13747
(https://issues.apache.org/jira/secure/attachment/12835043/benchmark_results)
which indicate that this can bring significantly higher performance in high
contention situations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13583) Incorporate checkcompatibility script which runs Java API Compliance Checker
[ https://issues.apache.org/jira/browse/HADOOP-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626673#comment-15626673 ] Robert Kanter commented on HADOOP-13583: +1 after these two trivial things: - 2 lines with tabs reported by Jenkins - checkcompatibility.py should have executable permissions like the other programs in the bin dir, and so you can actually run it out of the box :) > Incorporate checkcompatibility script which runs Java API Compliance Checker > > > Key: HADOOP-13583 > URL: https://issues.apache.org/jira/browse/HADOOP-13583 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HADOOP-13583.001.patch, HADOOP-13583.002.patch, > HADOOP-13583.003.patch > > > Based on discussion at YETUS-445, this code can't go there, but it's still > very useful for release managers. A similar variant of this script has been > used for a while by Apache HBase and Apache Kudu, and IMO JACC output is > easier to understand than JDiff. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Status: Patch Available (was: Open)
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13427) Eliminate needless uses of FileSystem.exists, isFile, isDirectory
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626656#comment-15626656
]
Hadoop QA commented on HADOOP-13427:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-13427 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13427 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830700/HADOOP-13427-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10949/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Eliminate needless uses of FileSystem.exists, isFile, isDirectory
> --
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13427) Eliminate needless uses of FileSystem.exists, isFile, isDirectory
[ https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13427: --- Status: Patch Available (was: Open) > Eliminate needless uses of FileSystem.exists, isFile, isDirectory > -- > > Key: HADOOP-13427 > URL: https://issues.apache.org/jira/browse/HADOOP-13427 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch > > > We're cleaning up Hive and Spark's use of FileSystem.exists, because it is > often the case we see code of exists+open, exists+delete, when the exists > probe is needless. Against object stores, expensive needless. > Hadoop can set an example here by stripping them out. It will also show where > there are opportunities to optimise things better and/or improve reporting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13667) Fix typing mistake of inline document in hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626570#comment-15626570
]
Daniel Templeton commented on HADOOP-13667:
---
Thanks for the patch, [~demongaorui]. Looks good to me, but I don't know much
about exporting the metrics to Ganglia. [~andrew.wang], can you take a look or
recommend a reviewer?
> Fix typing mistake of inline document in hadoop-metrics2.properties
> ---
>
> Key: HADOOP-13667
> URL: https://issues.apache.org/jira/browse/HADOOP-13667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Rui Gao
>Assignee: Rui Gao
> Attachments: HADOOP-13667.2.patch, HADOOP-13667.patch
>
>
> Fix typing mistake of inline document in hadoop-metrics2.properties.
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcesName{code} should be
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcessName{code}
> And also could add examples into the inline document for easier understanding
> of metrics tag related configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13453: --- Assignee: Ai Deng > S3Guard: Instrument new functionality with Hadoop metrics. > -- > > Key: HADOOP-13453 > URL: https://issues.apache.org/jira/browse/HADOOP-13453 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Ai Deng > > Provide Hadoop metrics showing operational details of the S3Guard > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8500) Fix javadoc jars to not contain entire target directory
[
https://issues.apache.org/jira/browse/HADOOP-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626465#comment-15626465
]
Allen Wittenauer commented on HADOOP-8500:
--
{code}
$ git status
On branch h13397
Untracked files:
(use "git add ..." to include in what will be committed)
hadoop-common-project/hadoop-common/api/
hadoop-hdfs-project/hadoop-hdfs-client/api/
hadoop-hdfs-project/hadoop-hdfs/api/
{code}
:(
> Fix javadoc jars to not contain entire target directory
> ---
>
> Key: HADOOP-8500
> URL: https://issues.apache.org/jira/browse/HADOOP-8500
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: N/A
>Reporter: EJ Ciramella
>Assignee: Andrew Wang
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-8500.001.patch, HADOOP-8500.002.patch,
> HADOOP-8500.003.patch, HADOOP-8500.patch, site-redo.tar
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The javadoc jars contain the contents of the target directory - which
> includes classes and all sorts of binary files that it shouldn't.
> Sometimes the resulting javadoc jar is 10X bigger than it should be.
> The fix is to reconfigure maven to use "api" as it's destDir for javadoc
> generation.
> I have a patch/diff incoming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13632) Daemonization does not check process liveness before renicing
[
https://issues.apache.org/jira/browse/HADOOP-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626450#comment-15626450
]
Allen Wittenauer commented on HADOOP-13632:
---
The hadoop_status_daemon check in hadoop_start_secure_daemon_wrapper is
checking the wrong pid file. $pidfile hasn't been defined.
> Daemonization does not check process liveness before renicing
> -
>
> Key: HADOOP-13632
> URL: https://issues.apache.org/jira/browse/HADOOP-13632
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13632.001.patch
>
>
> If you try to daemonize a process that is incorrectly configured, it will die
> quite quickly. However, the daemonization function will still try to renice
> it even if it's down, leading to something like this for my namenode:
> {noformat}
> -> % bin/hdfs --daemon start namenode
> ERROR: Cannot set priority of namenode process 12036
> {noformat}
> It'd be more user-friendly instead of this renice error, we said that the
> process couldn't be started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626431#comment-15626431
]
Allen Wittenauer commented on HADOOP-13717:
---
+1
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626420#comment-15626420 ] Larry McCay commented on HADOOP-12804: -- [~ste...@apache.org], [~cnauroth] - can I get a review for this issue patch? Thanks! > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-004.patch, HADOOP-12804-005.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626413#comment-15626413
]
Allen Wittenauer commented on HADOOP-13771:
---
What I had in my head is that if 'hdfs groups' can't contact the NN after a
much shorter timeout, it would then run locally and provide an answer with the
additional text of "(non-authoritative)" or something else in the output.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.patch
>
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13771) Adding group mapping lookup utility without dependency on HDFS namenode
[
https://issues.apache.org/jira/browse/HADOOP-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626392#comment-15626392
]
Xiaoyu Yao commented on HADOOP-13771:
-
Thanks [~aw] for providing the details of "hdfs groups". That well explained
why we have "hdfs groups" instead of "hadoop groups" today.
bq. There's an additional wrinkle here. The NN is not the only process that is
doing group resolution. Pretty much any service that does ACL resolution also
does group resolution to some degree. Making the command 'hadoop groups' is
going lead some folks to think that this works for any service...
Agree. Based on your description, I would prefer keep "hdfs groups" as-is today
instead of replacing "hdfs groups".
How about expose this as a DEBUG tool only? Below are some choices here:
1) run only with class Main only, no CLI exposed.
hadoop org.apache.hadoop.security.Groups
2) Add "hadoop groups" which wraps 1) in script, less ideal as you mentioned
above.
3) Add "hdfs debug groups" which wrapps 1) in script.
Explicitly mention the result is solely based on the configurations from
core-site.xml configurations.
It is authoritative compared with "hdfs groups"
bq. I'd therefore propose a different solution. 'hdfs groups' should work like
nslookup. If the NN is up, it should query the NN and give an authoritative
answer. If the NN is not up, it should give the local answer but be absolutely
clear that it is at best a guess and may be in correct.
This proposal looks good to me as well. MR and HDFS share a common base for the
"group" lookup. This will change the group lookup for both HDFS and MR.
> Adding group mapping lookup utility without dependency on HDFS namenode
> ---
>
> Key: HADOOP-13771
> URL: https://issues.apache.org/jira/browse/HADOOP-13771
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, tools
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-13771.00.patch
>
>
> We have {{hdfs groups}} command to troubleshoot issues related to users'
> group member look up with Unix/LDAP. However, there are some limitation of
> this command: 1) it can only be executed when namenode is running. 2) any
> change in the group mapping lookup configuration needs a hdfs namenode
> restart, which is expensive.
> This ticket is proposed to have a simple CLI utility like HadoopKerberosName
> {code}
> hadoop org.apache.hadoop.security.HadoopKerberosName
> nn/localh...@hdpdev.dev.com
> {code}
> The CLI utility for group member lookup will have a usage like below without
> namenode running or restart for configuration change.
> {code}
> hadoop org.apache.hadoop.security.Groups hdfs
> hdfs : [hadoop, hdfs]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.patch
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Description:
YARN-4243 introduced the logic that lets retry establishing the connection when
initializing the {{ActiveStandbyElector}} adding the parameter {{failFast}}.
{{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
set to false, to let the ZFKC wait longer for the Zookeeper server be in a
ready state when first initializing it.
was:
YARN-4243 introduced the logic that lets retry establishing the connection when
initializing the `ActiveStandbyElector` adding the parameter `failFast`.
`ZKFailoverController#initZK` should use this constructor with `failFast` set
to false, to let the ZFKC wait longer for the Zookeeper server be in a ready
state when first initializing it.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13673) Update sbin/start-* and sbin/stop-* to be smarter
[ https://issues.apache.org/jira/browse/HADOOP-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626353#comment-15626353 ] Allen Wittenauer commented on HADOOP-13673: --- Argh. I'll fix the hadoop-project/pom.xml issue on 01. :( > Update sbin/start-* and sbin/stop-* to be smarter > - > > Key: HADOOP-13673 > URL: https://issues.apache.org/jira/browse/HADOOP-13673 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13673.00.patch > > > As work continues on HADOOP-13397, it's become evident that we need better > hooks to start daemons as specifically configured users. Via the > (command)_(subcommand)_USER environment variables in 3.x, we actually have a > standardized way to do that. This in turn means we can make the sbin scripts > super functional with a bit of updating: > * Consolidate start-dfs.sh and start-secure-dns.sh into one script > * Make start-\*.sh and stop-\*.sh know how to switch users when run as root > * Undeprecate start/stop-all.sh so that it could be used as root for > production purposes and as a single user for non-production users -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
Andres Perez created HADOOP-13781: - Summary: ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false Key: HADOOP-13781 URL: https://issues.apache.org/jira/browse/HADOOP-13781 Project: Hadoop Common Issue Type: Improvement Components: ha Affects Versions: 3.0.0-alpha1 Reporter: Andres Perez Priority: Minor YARN-4243 introduced the logic that lets retry establishing the connection when initializing the `ActiveStandbyElector` adding the parameter `failFast`. `ZKFailoverController#initZK` should use this constructor with `failFast` set to false, to let the ZFKC wait longer for the Zookeeper server be in a ready state when first initializing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13673) Update sbin/start-* and sbin/stop-* to be smarter
[ https://issues.apache.org/jira/browse/HADOOP-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13673: -- Status: Patch Available (was: Open) > Update sbin/start-* and sbin/stop-* to be smarter > - > > Key: HADOOP-13673 > URL: https://issues.apache.org/jira/browse/HADOOP-13673 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13673.00.patch > > > As work continues on HADOOP-13397, it's become evident that we need better > hooks to start daemons as specifically configured users. Via the > (command)_(subcommand)_USER environment variables in 3.x, we actually have a > standardized way to do that. This in turn means we can make the sbin scripts > super functional with a bit of updating: > * Consolidate start-dfs.sh and start-secure-dns.sh into one script > * Make start-\*.sh and stop-\*.sh know how to switch users when run as root > * Undeprecate start/stop-all.sh so that it could be used as root for > production purposes and as a single user for non-production users -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13673) Update sbin/start-* and sbin/stop-* to be smarter
[ https://issues.apache.org/jira/browse/HADOOP-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626343#comment-15626343 ] Allen Wittenauer edited comment on HADOOP-13673 at 11/1/16 7:00 PM: -00: * first pass Running the start-* and stop-* commands will fire off daemons either as the user they are being run as or if the effective user id is root, as the appropriate _user definition. Secure daemons will "do the right thing"--get started as root but then switch to the appropriate user when needed. At this point, the old start-secure and start-dfs are not merged. I may do that in a future pass. was (Author: aw): -00: * first pass Running the start-* and stop-* commands will fire off daemons either as the user they are being run as or if the effective user id is root, as the appropriate _user definition. Secure daemons will "do the right thing"--get started as root but then switch to the appropriate user when needed. > Update sbin/start-* and sbin/stop-* to be smarter > - > > Key: HADOOP-13673 > URL: https://issues.apache.org/jira/browse/HADOOP-13673 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13673.00.patch > > > As work continues on HADOOP-13397, it's become evident that we need better > hooks to start daemons as specifically configured users. Via the > (command)_(subcommand)_USER environment variables in 3.x, we actually have a > standardized way to do that. This in turn means we can make the sbin scripts > super functional with a bit of updating: > * Consolidate start-dfs.sh and start-secure-dns.sh into one script > * Make start-\*.sh and stop-\*.sh know how to switch users when run as root > * Undeprecate start/stop-all.sh so that it could be used as root for > production purposes and as a single user for non-production users -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13773: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Resolving since it looks like this was committed to branch-2.8. Also it looks like this was committed without a precommit run. Let's try to be more careful in the future. > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Fix For: 2.8.0 > > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[
https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626318#comment-15626318
]
Hadoop QA commented on HADOOP-13773:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red} HADOOP-13773 does not apply to branch-2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13773 |
| GITHUB PR | https://github.com/apache/hadoop/pull/150 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10947/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
> ---
>
> Key: HADOOP-13773
> URL: https://issues.apache.org/jira/browse/HADOOP-13773
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.6.1, 2.7.3
>Reporter: Fei Hui
>Assignee: Fei Hui
> Fix For: 2.8.0
>
> Attachments: HADOOP-13773.patch
>
>
> in conf/hadoop-env.sh,
> export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
> when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work.
> i see, in bin/hadoop,
> exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"
> HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work.
> for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is
> 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is
> invalid
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13673) Update sbin/start-* and sbin/stop-* to be smarter
[ https://issues.apache.org/jira/browse/HADOOP-13673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13673: -- Attachment: HADOOP-13673.00.patch -00: * first pass Running the start-* and stop-* commands will fire off daemons either as the user they are being run as or if the effective user id is root, as the appropriate _user definition. Secure daemons will "do the right thing"--get started as root but then switch to the appropriate user when needed. > Update sbin/start-* and sbin/stop-* to be smarter > - > > Key: HADOOP-13673 > URL: https://issues.apache.org/jira/browse/HADOOP-13673 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13673.00.patch > > > As work continues on HADOOP-13397, it's become evident that we need better > hooks to start daemons as specifically configured users. Via the > (command)_(subcommand)_USER environment variables in 3.x, we actually have a > standardized way to do that. This in turn means we can make the sbin scripts > super functional with a bit of updating: > * Consolidate start-dfs.sh and start-secure-dns.sh into one script > * Make start-\*.sh and stop-\*.sh know how to switch users when run as root > * Undeprecate start/stop-all.sh so that it could be used as root for > production purposes and as a single user for non-production users -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626288#comment-15626288
]
Sangjin Lee commented on HADOOP-11804:
--
Thanks for the work [~busbey]! I just did a quick test with the latest patch.
One high level concern is in terms of maintaining dependencies in the pom's. If
a developer adds a new dependency to a module, how would that propagate to
these client pom's? Would he/she need to add it to these client pom's for the
most part? It wasn't entirely clear to me what that cost of maintenance is. If
that is the only way to keep it clean, that's OK. But it would be great if that
cost is kept to a minimum.
1.
The patch indeed does not apply for me via plain {{git apply}}: it breaks with
{{hadoop-client/pom.xml}} and {{hadoop-maven-plugins/pom.xml}}. I did {{git
apply --reject HADOOP-11804.1.patch}}.
2.
Once I fixed the git apply issues, I did {{mvn clean install package -Pdist
-DskipTests -Dmaven.javadoc.skip}} and it fails right away:
{noformat}
[ERROR] The project
org.apache.hadoop:hadoop-client-minicluster:3.0.0-alpha2-SNAPSHOT
(/Users/sjlee/git/hadoop-trunk/hadoop-client-modules/hadoop-client-minicluster/pom.xml)
has 1 error
[ERROR] 'dependencies.dependency.version' for org.mortbay.jetty:jetty:jar
is missing. @ line 266, column 17
{noformat}
I got past it by providing a version for this (chose 6.1.26).
3.
The build still fails with a couple of duplicate classes issues. One is what
Andrew reported above. Another is duplicate jetty classes.
{noformat}
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/org/eclipse/jetty/io/ssl/SslConnection$2.class
org/apache/hadoop/shaded/org/eclipse/jetty/server/RequestLog.class
org/apache/hadoop/shaded/org/eclipse/jetty/server/ResourceCache$1.class
org/apache/hadoop/shaded/org/eclipse/jetty/util/log/AbstractLogger.class
org/apache/hadoop/shaded/org/eclipse/jetty/util/annotation/Name.class
org/apache/hadoop/shaded/org/eclipse/jetty/util/component/LifeCycle.class
org/apache/hadoop/shaded/org/eclipse/jetty/server/HttpChannel$Commit100Callback.class
org/apache/hadoop/shaded/org/eclipse/jetty/util/ssl/SslContextFactory$1.class
...
{noformat}
4.
Was there a significant difficulty in handing the timeline service v.2? Is it
just the number of new dependencies we’re pulling in or the fact that there is
a HBase dependency?
5.
Regarding the logging libraries, I agree we probably want to exclude them.
Things like log4j properties and the way slf4j works can cause issues down the
road if shaded.
> POC Hadoop Client w/o transitive dependencies
> -
>
> Key: HADOOP-11804
> URL: https://issues.apache.org/jira/browse/HADOOP-11804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch,
> HADOOP-11804.3.patch, HADOOP-11804.4.patch
>
>
> make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to
> talk with a Hadoop cluster without seeing any of the implementation
> dependencies.
> see proposal on parent for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13738) DiskChecker should perform some disk IO
[ https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626274#comment-15626274 ] Xiaoyu Yao commented on HADOOP-13738: - Thanks [~arpitagarwal] for updating the patch. Patch v5 LGTM. +1. > DiskChecker should perform some disk IO > --- > > Key: HADOOP-13738 > URL: https://issues.apache.org/jira/browse/HADOOP-13738 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch, > HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch > > > DiskChecker can fail to detect total disk/controller failures indefinitely. > We have seen this in real clusters. DiskChecker performs simple > permissions-based checks on directories which do not guarantee that any disk > IO will be attempted. > A simple improvement is to write some data and flush it to the disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626245#comment-15626245
]
Lei (Eddy) Xu commented on HADOOP-13650:
Thanks for the referring to the patch, [~ste...@apache.org]. I will revise the
code based on HADOOP-13311.
For the status of this patch, the patch is almost ready, mostly waiting for
dynamodb metadata store being committed.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13773: -- Status: Patch Available (was: Open) > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.7.3, 2.6.1 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626155#comment-15626155 ] Ravi Prakash commented on HADOOP-13773: --- Just for future reference. I've added you to the Contributors1 role. That would allow you to assign issues to yourself. Also, please follow the patch naming scheme. All patch files should have a version. Also, if the patch is not for trunk, it should contain the branch name. So the file name for your patch should be HADOOP-13773.branch-2.01.patch. Thanks for your contribution and we look forward to many more from you! > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-13773: -- Assignee: Fei Hui > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13773) wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2
[ https://issues.apache.org/jira/browse/HADOOP-13773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626142#comment-15626142 ] Ravi Prakash commented on HADOOP-13773: --- Thanks for your contribution Fei Hui and for your careful review Yuanbo Liu! I've committed this to branch-2 and branch-2.8. > wrong HADOOP_CLIENT_OPTS in hadoop-env on branch-2 > --- > > Key: HADOOP-13773 > URL: https://issues.apache.org/jira/browse/HADOOP-13773 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.6.1, 2.7.3 >Reporter: Fei Hui > Attachments: HADOOP-13773.patch > > > in conf/hadoop-env.sh, > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > when i set HADOOP_HEAPSIZE ,and run 'hadoop jar ...', jvm args is not work. > i see, in bin/hadoop, > exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" > HADOOP_OPTS is behind JAVA_HEAP_MAX, so HADOOP_HEAPSIZE is not work. > for example i run 'HADOOP_HEAPSIZE=1024 hadoop jar ...' , the java process is > 'java -Xmx1024m ... -Xmx512m...', then Xmx512m is valid， and Xmx1024m is > invalid -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11804) POC Hadoop Client w/o transitive dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626097#comment-15626097 ] Andrew Wang commented on HADOOP-11804: -- Thanks for the quick response Sean, bq. I'm confused about test protobuf classes showing up in the client-api jar... Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions. bq. I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here. The logging rationale here sounds good to me. We've been trying to migrate things to slf4j, but evidently we haven't made much progress. Unshading HTrace also sounds good, since I'm guessing that shading will mess up tracing from an app into the Hadoop client. [~cmccabe] care to comment more authoritatively? > POC Hadoop Client w/o transitive dependencies > - > > Key: HADOOP-11804 > URL: https://issues.apache.org/jira/browse/HADOOP-11804 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HADOOP-11804.1.patch, HADOOP-11804.2.patch, > HADOOP-11804.3.patch, HADOOP-11804.4.patch > > > make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to > talk with a Hadoop cluster without seeing any of the implementation > dependencies. > see proposal on parent for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[
https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625994#comment-15625994
]
Xiao Chen commented on HADOOP-13590:
Thanks [~ste...@apache.org] for the prompt review!
Good point on {{getMaxTgtRenewalRetryCount}}, on a second thought I think it
can be eliminated, so the retry policy goes to {{Int.MAX_VALUE}} and we simply
check it against the end time. Currently it's only making sure we can create
the RetryPolicy with correct maxRetries. Will do that in the next patch, and
add comments.
bq. Test-wise, I've added support for more backoff in tests that wait; look in
LambdaTestUtils.
Thanks for the good work, let me try replace the GenericTestUtil usage with it.
bq. I also see that the code to set up a
javax.security.auth.login.Configuration is surfacing again...
See my [comment
above|https://issues.apache.org/jira/browse/HADOOP-13590?focusedCommentId=15517201=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15517201],
it's due to conflicting class name {{Configuration}} in hadoop and in javax. I
guess we'll have to explicitly define one way or the other. :(
Happy to wrap up a utility function to clean up all IBM hacks etc., I propose
to create a separate jira to limit scope of this one. Please let me know if you
feel otherwise.
> Retry until TGT expires even if the UGI renewal thread encountered exception
>
>
> Key: HADOOP-13590
> URL: https://issues.apache.org/jira/browse/HADOOP-13590
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.4
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13590.01.patch, HADOOP-13590.02.patch,
> HADOOP-13590.03.patch, HADOOP-13590.04.patch, HADOOP-13590.05.patch,
> HADOOP-13590.06.patch, HADOOP-13590.07.patch, HADOOP-13590.08.patch
>
>
> The UGI has a background thread to renew the tgt. On exception, it
> [terminates
> itself|https://github.com/apache/hadoop/blob/bee9f57f5ca9f037ade932c6fd01b0dad47a1296/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1013-L1014]
> If something temporarily goes wrong that results in an IOE, even if it
> recovered no renewal will be done and client will eventually fail to
> authenticate. We should retry with our best effort, until tgt expires, in the
> hope that the error recovers before that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11229) JobStoryProducer is not closed upon return from Gridmix#setupDistCacheEmulation()
[
https://issues.apache.org/jira/browse/HADOOP-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625962#comment-15625962
]
Ted Yu commented on HADOOP-11229:
-
Agree with Tsuyoshi Ozawa.
> JobStoryProducer is not closed upon return from
> Gridmix#setupDistCacheEmulation()
> -
>
> Key: HADOOP-11229
> URL: https://issues.apache.org/jira/browse/HADOOP-11229
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: skrho
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11229.v3.patch, HADOOP-11229_001.patch,
> HADOOP-11229_002.patch
>
>
> Here is related code:
> {code}
> JobStoryProducer jsp = createJobStoryProducer(traceIn, conf);
> exitCode = distCacheEmulator.setupGenerateDistCacheData(jsp);
> {code}
> jsp should be closed upon return from setupDistCacheEmulation().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13309) Document S3A known limitations in file ownership and permission model.
[ https://issues.apache.org/jira/browse/HADOOP-13309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625952#comment-15625952 ] ASF GitHub Bot commented on HADOOP-13309: - Github user cnauroth closed the pull request at: https://github.com/apache/hadoop/pull/138 > Document S3A known limitations in file ownership and permission model. > -- > > Key: HADOOP-13309 > URL: https://issues.apache.org/jira/browse/HADOOP-13309 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > > S3A does not match the implementation of HDFS in its handling of file > ownership and permissions. Fundamental S3 limitations prevent it. This is a > frequent source of confusion for end users. This issue proposes to document > these known limitations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13729) switch to Configuration.getLongBytes for byte options
[
https://issues.apache.org/jira/browse/HADOOP-13729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13729.
-
Resolution: Duplicate
Assignee: Abhishek Modi (was: Steve Loughran)
Fix Version/s: 2.8.0
> switch to Configuration.getLongBytes for byte options
> -
>
> Key: HADOOP-13729
> URL: https://issues.apache.org/jira/browse/HADOOP-13729
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Abhishek Modi
> Fix For: 2.8.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> It's fiddly working out how many bytes to use for 128MB readahead, equally
> hard to work out what a value actually means.
> If we switch to {{Configuration.getLongBytes()}} for reading in readahead,
> partition and threshold values, all existing configs will work, but new
> configs can use K, M, G suffices.
> Easy to code; should add a new test/adapt existing ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13768) AliyunOSS: DeleteObjectsRequest has 1000 objects limit
[
https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625802#comment-15625802
]
Hadoop QA commented on HADOOP-13768:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13768 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836376/HADOOP-13768.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cabc03927a27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5b577f4 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10946/testReport/ |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10946/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: DeleteObjectsRequest has 1000 objects limit
> ---
>
> Key: HADOOP-13768
> URL: https://issues.apache.org/jira/browse/HADOOP-13768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[
https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625785#comment-15625785
]
Hadoop QA commented on HADOOP-13769:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
6s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13769 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836377/HADOOP-13769.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 4ddd58c3be67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5b577f4 |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10945/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10945/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS object inputstream.close() will read the remaining bytes of the OSS
> object, potentially transferring a lot of bytes from OSS that are discarded.
> --
>
> Key: HADOOP-13769
> URL: https://issues.apache.org/jira/browse/HADOOP-13769
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/oss
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13769.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands,
[jira] [Assigned] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu reassigned HADOOP-13769: -- Assignee: Genmao Yu > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. > -- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13769: --- Attachment: HADOOP-13769.001.patch Update: just upgrade the aliyun oss sdk version > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. > -- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13769: --- Status: Patch Available (was: Open) > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. > -- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13768) AliyunOSS: DeleteObjectsRequest has 1000 objects limit
[ https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13768: --- Attachment: HADOOP-13768.002.patch > AliyunOSS: DeleteObjectsRequest has 1000 objects limit > --- > > Key: HADOOP-13768 > URL: https://issues.apache.org/jira/browse/HADOOP-13768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
