[jira] [Commented] (MAPREDUCE-6508) TestNetworkedJob fails consistently due to delegation token changes on RM.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091469#comment-15091469
]
Naganarasimha G R commented on MAPREDUCE-6508:
--
Hi [~ajisakaa],
As i understand YARN-3946 broke it once, dint get what you meant by {{but after
MAPREDUCE-6508, YARN-3946 broke the test again}}. Am i missing something ?
> TestNetworkedJob fails consistently due to delegation token changes on RM.
> --
>
> Key: MAPREDUCE-6508
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6508
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6508.00.patch, MAPREDUCE-6508.01.patch
>
>
> {noformat}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 84.215 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 31.537 sec <<< ERROR!
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException:
> java.io.IOException: Delegation Token can be issued only with kerberos
> authentication
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1044)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getDelegationToken(ApplicationClientProtocolPBServiceImpl.java:325)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2236)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2232)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2230)
> Caused by: java.io.IOException: Delegation Token can be issued only with
> kerberos authentication
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getDelegationToken(ClientRMService.java:1017)
> ... 10 more
> at org.apache.hadoop.ipc.Client.call(Client.java:1448)
> at org.apache.hadoop.ipc.Client.call(Client.java:1379)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy84.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getDelegationToken(ApplicationClientProtocolPBClientImpl.java:339)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy85.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getRMDelegationToken(YarnClientImpl.java:541)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getDelegationToken(ResourceMgrDelegate.java:177)
> at
> org.apache.hadoop.mapred.YARNRunner.getDelegationToken(YARNRunner.java:231)
> at
> org.apache.hadoop.mapreduce.Cluster.getDelegationToken(Cluster.java:401)
> at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1234)
> at org.apache.hadoop.mapred.JobClient$16.run(JobClient.java:1231)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at
> org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:1230)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:260)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091378#comment-15091378
]
Hadoop QA commented on MAPREDUCE-6068:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 15s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
(total was 71, now 72). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 2s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 41s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781488/MAPREDUCE-6068.003.patch
|
| JIRA Issue | MAPREDUCE-6068 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Updated] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated MAPREDUCE-6068:
-
Attachment: MAPREDUCE-6068.003.patch
Thanks for the review and nice catch [~djp] ! Update the patch.
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.003.patch,
> MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6602) ApplicationResourceUsageReport should return ALL reserved resource
Bibin A Chundatt created MAPREDUCE-6602: --- Summary: ApplicationResourceUsageReport should return ALL reserved resource Key: MAPREDUCE-6602 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6602 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt ApplicationResourceUsageReport reserved resource report is only of default parition should be of all partitions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6068) Illegal progress value warnings in map tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091297#comment-15091297
]
Junping Du commented on MAPREDUCE-6068:
---
Forget to mention, there is still another place in Merger.java that could over
the limit:
{code}
if (totalBytes != 0)
mergeProgress.set(totalBytesProcessed * progPerByte);
else
mergeProgress.set(1.0f); // Last pass and no segments left - we're
done
{code}
[~decster], do you think we should update here also?
> Illegal progress value warnings in map tasks
>
>
> Key: MAPREDUCE-6068
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6068
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
>Affects Versions: 3.0.0
>Reporter: Todd Lipcon
>Assignee: Binglin Chang
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6068.002.patch, MAPREDUCE-6068.v1.patch
>
>
> When running a terasort on latest trunk, I see the following in my task logs:
> {code}
> 2014-09-02 17:42:28,437 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,238 WARN [main] org.apache.hadoop.util.Progress: Illegal
> progress value found, progress is larger than 1. Progress will be changed to 1
> 2014-09-02 17:42:42,241 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> {code}
> We should eliminate these warnings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
