[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443525#comment-16443525
]
genericqa commented on HADOOP-15129:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 174 unchanged - 0 fixed = 175 total (was 174) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
30s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15129 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905521/HADOOP-15129.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e97bda07cb2e 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e4c39f3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14503/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14503/testReport/ |
| Max. process+thread count | 1767 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14503/console |
| Powered by | Apache Yetus
[jira] [Commented] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
[
https://issues.apache.org/jira/browse/HADOOP-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443482#comment-16443482
]
genericqa commented on HADOOP-15390:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
41s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15390 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919702/HADOOP-15390.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ac0de2cfc4bc 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bf7694d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443428#comment-16443428
]
Karthik Palaniappan commented on HADOOP-15129:
--
I forgot about this for a while – is there anything blocking merging the patch?
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
[
https://issues.apache.org/jira/browse/HADOOP-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HADOOP-15390:
-
Priority: Critical (was: Major)
Target Version/s: 3.1.1
Can you please add which patch originally caused this and which versions are
affected.
Tentatively putting target-version as 3.1.1 till then.
> Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
> -
>
> Key: HADOOP-15390
> URL: https://issues.apache.org/jira/browse/HADOOP-15390
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HADOOP-15390.01.patch, HADOOP-15390.02.patch
>
>
> When looking at a recent issue with [~rkanter] and [~yufeigu], we found that
> the RM log in a cluster was flooded by KMS token renewal errors below:
> {noformat}
> $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> ...
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> {noformat}
> Further inspection shows the KMS IP is from another cluster. The RM is before
> HADOOP-14445, so needs to read from config. The config rightfully doesn't
> have the other cluster's KMS configured.
> Although HADOOP-14445 will make this a non-issue by creating the provider
> from token service, we should fix 2 things here:
> - KMS token renewer should throw instead of return 0. Returning 0 when not
> able to renew shall be considered a bug in the renewer.
> - Yarn RM's {{DelegationTokenRenewer}} service should validate the return and
> not go into this busy loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15396:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.5
3.0.3
2.9.2
3.1.1
3.2.0
2.10.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-3.1, branch-3.0, branch-2, branch-2.9, and
branch-2.8. Thanks [~shashikant] for the contribution!
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.8.5
>
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
[
https://issues.apache.org/jira/browse/HADOOP-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443316#comment-16443316
]
Xiao Chen commented on HADOOP-15390:
Thanks for the review Robert.
bq. 1
Put all details into the exception. Removed the logging part as that's not
necessary anymore.
bq. 2
That's "Added a dummy line in TestKMS for pre-commit coverage.", because I want
hadoop-kms tests to be ran here. Above pre-commit gave a green, so removed it
in this patch. :)
> Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
> -
>
> Key: HADOOP-15390
> URL: https://issues.apache.org/jira/browse/HADOOP-15390
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15390.01.patch, HADOOP-15390.02.patch
>
>
> When looking at a recent issue with [~rkanter] and [~yufeigu], we found that
> the RM log in a cluster was flooded by KMS token renewal errors below:
> {noformat}
> $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> ...
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> {noformat}
> Further inspection shows the KMS IP is from another cluster. The RM is before
> HADOOP-14445, so needs to read from config. The config rightfully doesn't
> have the other cluster's KMS configured.
> Although HADOOP-14445 will make this a non-issue by creating the provider
> from token service, we should fix 2 things here:
> - KMS token renewer should throw instead of return 0. Returning 0 when not
> able to renew shall be considered a bug in the renewer.
> - Yarn RM's {{DelegationTokenRenewer}} service should validate the return and
> not go into this busy loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
[
https://issues.apache.org/jira/browse/HADOOP-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15390:
---
Attachment: HADOOP-15390.02.patch
> Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
> -
>
> Key: HADOOP-15390
> URL: https://issues.apache.org/jira/browse/HADOOP-15390
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15390.01.patch, HADOOP-15390.02.patch
>
>
> When looking at a recent issue with [~rkanter] and [~yufeigu], we found that
> the RM log in a cluster was flooded by KMS token renewal errors below:
> {noformat}
> $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> ...
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> {noformat}
> Further inspection shows the KMS IP is from another cluster. The RM is before
> HADOOP-14445, so needs to read from config. The config rightfully doesn't
> have the other cluster's KMS configured.
> Although HADOOP-14445 will make this a non-issue by creating the provider
> from token service, we should fix 2 things here:
> - KMS token renewer should throw instead of return 0. Returning 0 when not
> able to renew shall be considered a bug in the renewer.
> - Yarn RM's {{DelegationTokenRenewer}} service should validate the return and
> not go into this busy loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15390) Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
[
https://issues.apache.org/jira/browse/HADOOP-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443228#comment-16443228
]
Robert Kanter commented on HADOOP-15390:
Thanks for the patch [~xiaochen]. Two things:
# When throwing the {{IOException}}, why not also include the token details in
the message? The patch is currently only logging that part.
# Adding the "// test" command to {{TestKMS}} seems unnecessary?
> Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens
> -
>
> Key: HADOOP-15390
> URL: https://issues.apache.org/jira/browse/HADOOP-15390
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15390.01.patch
>
>
> When looking at a recent issue with [~rkanter] and [~yufeigu], we found that
> the RM log in a cluster was flooded by KMS token renewal errors below:
> {noformat}
> $ tail -9 hadoop-cmf-yarn-RESOURCEMANAGER.log
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> ...
> 2018-04-11 11:34:09,367 WARN
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer:
> keyProvider null cannot renew dt.
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renewed delegation-token= [Kind: kms-dt, Service: KMSIP:16000, Ident:
> (kms-dt owner=user, renewer=yarn, realUser=, issueDate=1522192283334,
> maxDate=1522797083334, sequenceNumber=15108613, masterKeyId=2674);exp=0;
> apps=[]], for []
> 2018-04-11 11:34:09,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
> Renew Kind: kms-dt, Service: KMSIP:16000, Ident: (kms-dt owner=user,
> renewer=yarn, realUser=, issueDate=1522192283334, maxDate=1522797083334,
> sequenceNumber=15108613, masterKeyId=2674);exp=0; apps=[] in -1523446449367
> ms, appId = []
> {noformat}
> Further inspection shows the KMS IP is from another cluster. The RM is before
> HADOOP-14445, so needs to read from config. The config rightfully doesn't
> have the other cluster's KMS configured.
> Although HADOOP-14445 will make this a non-issue by creating the provider
> from token service, we should fix 2 things here:
> - KMS token renewer should throw instead of return 0. Returning 0 when not
> able to renew shall be considered a bug in the renewer.
> - Yarn RM's {{DelegationTokenRenewer}} service should validate the return and
> not go into this busy loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah resolved HADOOP-15399.
-
Resolution: Invalid
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443064#comment-16443064
]
Rushabh S Shah commented on HADOOP-15399:
-
Thanks [~jojochuang] for jira pointers.
Will test and close if necessary.
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443030#comment-16443030
]
genericqa commented on HADOOP-15396:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
37s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 21s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
29s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
0s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}239m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15396 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919633/HADOOP-15396.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e4e25c8bac50
[jira] [Commented] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443020#comment-16443020
]
Wei-Chiu Chuang commented on HADOOP-15399:
--
Hi [~shahrs87],
hadoop.security.group.mapping related configs are loaded when KMS initializes
{code:title=KMSWebApp#contextInitialized}
kmsConf = KMSConfiguration.getKMSConf();
initLogging(confDir);
UserGroupInformation.setConfiguration(kmsConf);
{code}
(See: HADOOP-13638)
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Fix Version/s: 2.8.4
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Description:
KMSACLs uses {{AccessControlList}} for authorization.
For creating groups membership, the group implementation class that will be
instantiated is configured by {{hadoop.security.group.mapping}}.
Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
{{AccessControlList}}.
{{kms-acls.xml}} doesn't look the right place add the above config.
So KMSAcls should read either kms-site.
[~xiaochen]: Any preference which file should acls load ?
IMO it should be kms-site because that file is mandatory. But all the
properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit inclined
towards core-site.xml.
was:
KMSACLs create {{AccessControlList}} for authorization.
For creating groups membership, the group implementation class that will be
instantiated is configured by {{hadoop.security.group.mapping}}.
{{kms-acls.xml}} doesn't look the right place add the above config.
So KMSAcls should read either kms-site.
[~xiaochen]: Any preference which file should acls load ?
IMO it should be kms-site because that file is mandatory. But all the
properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit inclined
towards core-site.xml.
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
> Fix For: 2.8.4
>
>
> KMSACLs uses {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> Today {{KMSACLs}} class reads only {{kms-acls.xml}} file to create
> {{AccessControlList}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HADOOP-15399:
Component/s: kms
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
>
> KMSACLs create {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15399) KMSAcls should read kms-site.xml file.
[
https://issues.apache.org/jira/browse/HADOOP-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah moved YARN-8176 to HADOOP-15399:
---
Key: HADOOP-15399 (was: YARN-8176)
Project: Hadoop Common (was: Hadoop YARN)
> KMSAcls should read kms-site.xml file.
> --
>
> Key: HADOOP-15399
> URL: https://issues.apache.org/jira/browse/HADOOP-15399
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Major
>
> KMSACLs create {{AccessControlList}} for authorization.
> For creating groups membership, the group implementation class that will be
> instantiated is configured by {{hadoop.security.group.mapping}}.
> {{kms-acls.xml}} doesn't look the right place add the above config.
> So KMSAcls should read either kms-site.
> [~xiaochen]: Any preference which file should acls load ?
> IMO it should be kms-site because that file is mandatory. But all the
> properties in kms-site.xml starts with {{hadoop.kms}}, I am little bit
> inclined towards core-site.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-15180:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.2
2.8.4
2.10.0
Status: Resolved (was: Patch Available)
Committed to branch-2,branch-2.9 and branch-2.8.
Thanks [~RANith] for contribution and thanks [~vinayrpet] for additional review.
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Fix For: 2.10.0, 2.8.4, 2.9.2
>
> Attachments: HADOOP-15180-branch-2-002.patch,
> HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442927#comment-16442927
]
Sean Mackrory commented on HADOOP-15392:
{quote}If there's no way to stop this happening them the metric wire-up has to
be optional; keep off by default.{quote}
+1 to that approach if it comes to it. My first thought is that the metrics
configuration should be known at the time we make that decision, and in theory
we could only wire it up if the s3a metrics source would ever actually connect
to any sink. I suspect metrics2 doesn't expose the APIs to enable us to do that
in a clean way, though. I would also think that because it's designed for
daemons, the behavior in the event no sinks are configured would not be to just
leak memory forever, so I wonder if there's something else we're doing that's
wrong that we can just fix.
> S3A Metrics in S3AInstrumentation Cause Memory Leaks
>
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Voyta
>Priority: Major
>
> While using HBase S3A Export Snapshot utility we started to experience memory
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately
> metrics are accumulated in this instance and memory grows without any limit.
>
> Expectation:
> * It would be nice to have an option to disable metrics completely as this
> is not needed for Export Snapshot utility.
> * Usage of S3AFileSystem should not contain any static object that can grow
> indefinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442890#comment-16442890
]
Brahma Reddy Battula commented on HADOOP-15180:
---
+1 on latest patch.will commit soon
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180-branch-2-002.patch,
> HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442784#comment-16442784 ] Steve Loughran commented on HADOOP-15392: - That Export Snapshot gets filesystems in two places * [Mapper|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L224]: created in setup, destroyed in cleanup. Life should be that of the mapper alone. * CLI entry point [doWork|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java#L915]. Again, close() at the end, which is more than I often no. I don't see any leaking of FS instances anywhere: nothing in the HBase code to point a finger at. If we are leaking memory in the metrics is because the thing uses lots of memory *and* the ref counting isn't working. # Is there a config option we can set up to not accrue stuff by default? # maybe it's the fact that this is running with HBase configs which is making it surface. @Krizek: what is your metrics config file here? # If there's no way to stop this happening them the metric wire-up has to be optional; keep off by default. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442769#comment-16442769 ] Steve Loughran commented on HADOOP-15392: - Thank's ted. I'll just be looking at it to see what's happening about closing filesystems. If the metrics uses up lots of memory unless all uses of FS instances are closed(), this is dangerously brittle. You could play games with weak references, of course, but as JMX registration is also involved, that may not be enough for the metrics to get GC'd. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442747#comment-16442747
]
Akira Ajisaka commented on HADOOP-15396:
LGTM, +1.
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15398) Compilation error in trunk in hadoop-aws
[
https://issues.apache.org/jira/browse/HADOOP-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe moved HDFS-13472 to HADOOP-15398:
Target Version/s: 3.2.0, 3.1.1 (was: 3.2.0)
Key: HADOOP-15398 (was: HDFS-13472)
Project: Hadoop Common (was: Hadoop HDFS)
> Compilation error in trunk in hadoop-aws
> -
>
> Key: HADOOP-15398
> URL: https://issues.apache.org/jira/browse/HADOOP-15398
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Priority: Major
>
> *Problem:* hadoop trunk compilation is failing
> *Root Cause:*
> compilation error is coming from
> {{org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase}}. Compilation
> error is "The method getArgumentAt(int, Class) is
> undefined for the type InvocationOnMock".
> StagingTestBase is using getArgumentAt(int, Class) method
> which is not available in mockito-all 1.8.5 version. getArgumentAt(int,
> Class) method is available only from version 2.0.0-beta
> *Expectations:*
> Either mockito-all version to be upgraded or test case to be written only
> with available functions in 1.8.5.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442696#comment-16442696 ] Ted Yu commented on HADOOP-15392: - hbase export tool is located at: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java I don't see any metrics call in that utility. The code referenced in description is in S3AInstrumentation. So it seems this is specific to S3A code. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442652#comment-16442652 ] Steve Loughran commented on HADOOP-15392: - This isn't good. Looking @ the patch, the system is being shutdown & deleted when S3AInstrumentation.close() is called, and that is invoked when S3AFileSystem.close() is called. If only one instance in an FS is cached, then memory use is a problem if this metrics instance is storing lots and lots of stuff without cleanup. When the #of closed FS instances == 0, then the metrics system should be shut down (it's ref counted). If the fs.close() methods aren't being invoked, we could end up with the instances continuing being registered. Otherwise, maybe the cached entries aren't being cleaned up. Thoughts: [~fabbri] [~iyonger] [~sean_impala_9b93]? [~te...@apache.org] Ted: can you point me at HBase the code here? I'd like to see what it's doing. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442621#comment-16442621 ] Steve Loughran commented on HADOOP-15392: - HADOOP-14475 is the issue here. > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442619#comment-16442619
]
Shashikant Banerjee commented on HADOOP-15396:
--
patch vo fixes the permission for the java files with wrong permissions.
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15396:
-
Attachment: HADOOP-15396.000.patch
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HADOOP-15396:
-
Status: Patch Available (was: Open)
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15396.000.patch
>
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks
[ https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15392: Issue Type: Sub-task (was: Bug) Parent: HADOOP-15220 > S3A Metrics in S3AInstrumentation Cause Memory Leaks > > > Key: HADOOP-15392 > URL: https://issues.apache.org/jira/browse/HADOOP-15392 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Voyta >Priority: Major > > While using HBase S3A Export Snapshot utility we started to experience memory > leaks of the process after version upgrade. > By running code analysis we traced the cause to revision > 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static > reference (singleton): > private static MetricsSystem metricsSystem = null; > When application uses S3AFileSystem instance that is not closed immediately > metrics are accumulated in this instance and memory grows without any limit. > > Expectation: > * It would be nice to have an option to disable metrics completely as this > is not needed for Export Snapshot utility. > * Usage of S3AFileSystem should not contain any static object that can grow > indefinitely. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HADOOP-15396:
Assignee: Shashikant Banerjee
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Shashikant Banerjee
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11640) add user defined delimiter support to Configuration
[
https://issues.apache.org/jira/browse/HADOOP-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442593#comment-16442593
]
Jim Brennan commented on HADOOP-11640:
--
[~chris.douglas], thanks for the reply! [MAPREDUCE-7069] does not directly
address the Configuration.getStrings() delimiter issue.
It is more narrowly focussed on properties that define environment variables.
It provides a way of specifying them individually, thereby allowing for values
with commas in them.
It wasn't clear to me what the underlying motivation for this Jira was - I
thought it was possible that the motivation was to support environment variable
properties with commas in them, in which case [MAPREDUCE-7069] would be
relevant, and so I posted the question.
> add user defined delimiter support to Configuration
> ---
>
> Key: HADOOP-11640
> URL: https://issues.apache.org/jira/browse/HADOOP-11640
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Xiaoshuang LU
>Assignee: Xiaoshuang LU
>Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11640.patch
>
>
> As mentioned by org.apache.hadoop.conf.Configuration.getStrings ("Get the
> comma delimited values of the name property as an array of Strings"), only
> comma separated strings can be used. It would be much better if user defined
> separators are supported.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15366) Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
[
https://issues.apache.org/jira/browse/HADOOP-15366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HADOOP-15366:
---
Fix Version/s: 3.1.1
Cherry picked the fix to branch-3.1
> Add a helper shutdown routine in HadoopExecutor to ensure clean shutdown
>
>
> Key: HADOOP-15366
> URL: https://issues.apache.org/jira/browse/HADOOP-15366
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Minor
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15366.000.patch
>
>
> It is recommended to shut down an {{ExecutorService}} in two phases, first by
> calling {{shutdown}} to reject incoming tasks, and then calling
> {{shutdownNow}}, if necessary, to cancel any lingering tasks. This Jira aims
> to add a helper shutdown routine in Hadoop executor to achieve the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442249#comment-16442249
]
genericqa commented on HADOOP-15397:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 17m
25s{color} | {color:red} Docker failed to build yetus/hadoop:17213a0. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15397 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919600/HADOOP-15397-branch-2.9.0.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14500/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path,
> HADOOP-15397-branch-2.9.0.003.patch, HADOOP-15397.002.patch
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Attachment: HADOOP-15397-branch-2.9.0.003.patch
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path,
> HADOOP-15397-branch-2.9.0.003.patch, HADOOP-15397.002.patch
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442195#comment-16442195
]
genericqa commented on HADOOP-15397:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-15397 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15397 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919583/HADOOP-15397.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14499/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path, HADOOP-15397.002.patch
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-15396:
---
Assignee: Gabor Bota
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Gabor Bota
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota reassigned HADOOP-15396:
---
Assignee: (was: Gabor Bota)
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Attachment: HADOOP-15397.002.patch
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path, HADOOP-15397.002.patch
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442129#comment-16442129
]
genericqa commented on HADOOP-15397:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15397 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15397 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919574/HADOOP-15397-001.path
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14498/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Labels: (was: patch)
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Labels: patch (was: )
Status: Patch Available (was: Open)
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Labels: patch
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Attachment: (was: HADOOP-15397.path)
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Attachment: HADOOP-15397-001.path
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397-001.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Description:
You would get the following log, if you statt the estmator using script
start-estimator.sh;. And the estmator is not started.
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory{code}
Fix the bug in the script estimator-daemon.sh.
was:
You would get the following log, if you statt the estmator using script
start-estimator.sh;. And the estmator is not started.
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory{code}
> Fix the bug in the script estimator-daemon.sh.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[ https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HADOOP-15391: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.1 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-3.1. Thanks [~ajisakaa] for the review. > Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake modules > - > > Key: HADOOP-15391 > URL: https://issues.apache.org/jira/browse/HADOOP-15391 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg, > screen-shot.jpg > > > The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake render error (see screen-shot attached or > [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). > The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Attachment: HADOOP-15397.path
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
> Attachments: HADOOP-15397.path
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Description:
You would get the following log, if you statt the estmator using script
start-estimator.sh;. And the estmator is not started.
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
was:
You would get the following log, if you statt the estmator using script
start-estimator.sh:
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Minor
> Fix For: 2.9.0
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Priority: Major (was: Minor)
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Major
> Fix For: 2.9.0
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh;. And the estmator is not started.
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Description:
You would get the following log, if you statt the estmator using script
start-estimator.sh:
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
was:
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Minor
> Fix For: 2.9.0
>
>
> You would get the following log, if you statt the estmator using script
> start-estimator.sh:
>
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
[
https://issues.apache.org/jira/browse/HADOOP-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HADOOP-15397:
Description:
{code:java}
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
{code}
was:
starting resource estimator service
starting estimator, logging to
/hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
/hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47:
bin/estimator.sh: No such file or directory
> Failed to start the estimator of Resource Estimator Service
> ---
>
> Key: HADOOP-15397
> URL: https://issues.apache.org/jira/browse/HADOOP-15397
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.9.0
>Reporter: zhangbutao
>Priority: Minor
> Fix For: 2.9.0
>
>
> {code:java}
> starting resource estimator service
> starting estimator, logging to
> /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out
> /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line
> 47: bin/estimator.sh: No such file or directory
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15397) Failed to start the estimator of Resource Estimator Service
zhangbutao created HADOOP-15397: --- Summary: Failed to start the estimator of Resource Estimator Service Key: HADOOP-15397 URL: https://issues.apache.org/jira/browse/HADOOP-15397 Project: Hadoop Common Issue Type: Bug Components: tools Affects Versions: 2.9.0 Reporter: zhangbutao Fix For: 2.9.0 starting resource estimator service starting estimator, logging to /hadoop/share/hadoop/tools/resourceestimator/bin/../../../../../logs/hadoop-resourceestimator.out /hadoop/share/hadoop/tools/resourceestimator/bin/estimator-daemon.sh: line 47: bin/estimator.sh: No such file or directory -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[ https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442023#comment-16442023 ] Yiqun Lin commented on HADOOP-15391: Thanks for the review, [~ajisakaa]. The Jenkins looks good. Will commit shortly, :). > Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake modules > - > > Key: HADOOP-15391 > URL: https://issues.apache.org/jira/browse/HADOOP-15391 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg, > screen-shot.jpg > > > The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake render error (see screen-shot attached or > [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). > The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[
https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442016#comment-16442016
]
genericqa commented on HADOOP-15391:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
39m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15391 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919554/HADOOP-15391.001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 2a5ae03473e4 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d426b98 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure
hadoop-tools/hadoop-aliyun hadoop-tools/hadoop-azure-datalake U: hadoop-tools |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14497/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and
> hadoop-azure-datalake modules
> -
>
> Key: HADOOP-15391
> URL: https://issues.apache.org/jira/browse/HADOOP-15391
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg,
> screen-shot.jpg
>
>
> The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and
> hadoop-azure-datalake render error (see screen-shot attached or
> [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]).
> The reason of this is that the css file is missing in these modules.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15304) [JDK10] Migrate from com.sun.tools.doclets to the replacement
[
https://issues.apache.org/jira/browse/HADOOP-15304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442000#comment-16442000
]
Takanobu Asanuma commented on HADOOP-15304:
---
Oh, I thought it would remove the requirement ... anyway, thanks for caring it!
> [JDK10] Migrate from com.sun.tools.doclets to the replacement
> -
>
> Key: HADOOP-15304
> URL: https://issues.apache.org/jira/browse/HADOOP-15304
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15304.01.patch, HADOOP-15304.02.patch
>
>
> com.sun.tools.doclets.* packages were removed in Java 10.
> [https://bugs.openjdk.java.net/browse/JDK-8177511]
> This causes hadoop-annotations module to fail.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-annotations: Compilation failure: Compilation failure:
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsStandardDoclet.java:[61,20]
> cannot find symbol
> [ERROR] symbol: method
> validOptions(java.lang.String[][],com.sun.javadoc.DocErrorReporter)
> [ERROR] location: class com.sun.tools.doclets.standard.Standard
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java:[56,20]
> cannot find symbol
> [ERROR] symbol: method
> validOptions(java.lang.String[][],com.sun.javadoc.DocErrorReporter)
> [ERROR] location: class com.sun.tools.doclets.standard.Standard
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15393) Upgrade the version of commons-lang3 to 3.7
[
https://issues.apache.org/jira/browse/HADOOP-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441998#comment-16441998
]
Takanobu Asanuma commented on HADOOP-15393:
---
Thanks [~ajisakaa]!
> Upgrade the version of commons-lang3 to 3.7
> ---
>
> Key: HADOOP-15393
> URL: https://issues.apache.org/jira/browse/HADOOP-15393
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15393.1.patch
>
>
> {{mvn javadoc:javadoc}} with jdk10 fails due to MJAVADOC-517. This is
> reported by [~ajisakaa].
> Upgrading commons-lang3 version to the latest version 3.7 works round the
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[ https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441994#comment-16441994 ] Akira Ajisaka commented on HADOOP-15391: LGTM, +1 pending Jenkins. > Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake modules > - > > Key: HADOOP-15391 > URL: https://issues.apache.org/jira/browse/HADOOP-15391 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg, > screen-shot.jpg > > > The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake render error (see screen-shot attached or > [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). > The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[ https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HADOOP-15391: --- Attachment: HADOOP-15391.001.patch > Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake modules > - > > Key: HADOOP-15391 > URL: https://issues.apache.org/jira/browse/HADOOP-15391 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg, > screen-shot.jpg > > > The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake render error (see screen-shot attached or > [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). > The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
[ https://issues.apache.org/jira/browse/HADOOP-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HADOOP-15391: --- Attachment: (was: HADOOP-15391.001.patch) > Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake modules > - > > Key: HADOOP-15391 > URL: https://issues.apache.org/jira/browse/HADOOP-15391 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HADOOP-15391.001.patch, screen-shot-fixed.jpg, > screen-shot.jpg > > > The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and > hadoop-azure-datalake render error (see screen-shot attached or > [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). > The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15396:
---
Description:
{noformat}
$ find . -name "*.java" -perm 755
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
{noformat}
The permission should be 644. Let's fix them.
was:
{noformat}
$ find . -name "*.java" -perm 755
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
{noformat}
Let's fix them.
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> The permission should be 644. Let's fix them.
--
This message was sent by Atlassian JIRA
[jira] [Created] (HADOOP-15396) Some java source files are executable
Akira Ajisaka created HADOOP-15396:
--
Summary: Some java source files are executable
Key: HADOOP-15396
URL: https://issues.apache.org/jira/browse/HADOOP-15396
Project: Hadoop Common
Issue Type: Bug
Reporter: Akira Ajisaka
{noformat}
$ find . -name "*.java" -perm 755
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
{noformat}
Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15396) Some java source files are executable
[
https://issues.apache.org/jira/browse/HADOOP-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15396:
---
Labels: newbie (was: )
> Some java source files are executable
> -
>
> Key: HADOOP-15396
> URL: https://issues.apache.org/jira/browse/HADOOP-15396
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> {noformat}
> $ find . -name "*.java" -perm 755
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
> ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java
> ./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> {noformat}
> Let's fix them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15304) [JDK10] Migrate from com.sun.tools.doclets to the replacement
[
https://issues.apache.org/jira/browse/HADOOP-15304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15304:
---
Status: Patch Available (was: Open)
> [JDK10] Migrate from com.sun.tools.doclets to the replacement
> -
>
> Key: HADOOP-15304
> URL: https://issues.apache.org/jira/browse/HADOOP-15304
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15304.01.patch, HADOOP-15304.02.patch
>
>
> com.sun.tools.doclets.* packages were removed in Java 10.
> [https://bugs.openjdk.java.net/browse/JDK-8177511]
> This causes hadoop-annotations module to fail.
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-annotations: Compilation failure: Compilation failure:
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsStandardDoclet.java:[61,20]
> cannot find symbol
> [ERROR] symbol: method
> validOptions(java.lang.String[][],com.sun.javadoc.DocErrorReporter)
> [ERROR] location: class com.sun.tools.doclets.standard.Standard
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java:[56,20]
> cannot find symbol
> [ERROR] symbol: method
> validOptions(java.lang.String[][],com.sun.javadoc.DocErrorReporter)
> [ERROR] location: class com.sun.tools.doclets.standard.Standard
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
