[jira] [Updated] (MAPREDUCE-6386) Log Aggregation Deletion doesnt work after refreshing Log Retention Settings is invoked in secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6386: Priority: Critical (was: Major) Summary: Log Aggregation Deletion doesnt work after refreshing Log Retention Settings is invoked in secure cluster (was: refreshLogRetentionSettings is invalid in secure cluster) Log Aggregation Deletion doesnt work after refreshing Log Retention Settings is invoked in secure cluster - Key: MAPREDUCE-6386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6386 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Environment: mrV2, secure mode Reporter: Zhang Wei Assignee: Varun Saxena Priority: Critical bin/mapred hsadmin -refreshLogRetentionSettings is not available in secure cluster. reproduce steps: 1. startup historyserver in secure cluster. 2. login the secure user with kinit.(the same user with startup step) 3. execute bin/mapred hsadmin -refreshLogRetentionSettings to refresh. there will be exceptions as follows in the historyserver's log if log deletion is enabled. 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this deletion attempt is being aborted | AggregatedLogDeletionService.java:127 java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: vm-31/9.91.12.31; destination host is: vm-33:25000; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getListing(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462) at org.apache.hadoop.ipc.Client.call(Client.java:1381) ... 21 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any
[jira] [Commented] (MAPREDUCE-6386) Log Aggregation Deletion doesnt work after refreshing Log Retention Settings is invoked in secure cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574117#comment-14574117 ] Varun Saxena commented on MAPREDUCE-6386: - Able to simulate the issue. Deletion fails always (due to GSS API Exception) after refreshing log retention settings. Will look into it. Log Aggregation Deletion doesnt work after refreshing Log Retention Settings is invoked in secure cluster - Key: MAPREDUCE-6386 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6386 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Environment: mrV2, secure mode Reporter: Zhang Wei Assignee: Varun Saxena Priority: Critical bin/mapred hsadmin -refreshLogRetentionSettings is not available in secure cluster. reproduce steps: 1. startup historyserver in secure cluster. 2. login the secure user with kinit.(the same user with startup step) 3. execute bin/mapred hsadmin -refreshLogRetentionSettings to refresh. there will be exceptions as follows in the historyserver's log if log deletion is enabled. 2015-06-04 14:14:40,070 | ERROR | Timer-3 | Error reading root log dir this deletion attempt is being aborted | AggregatedLogDeletionService.java:127 java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: vm-31/9.91.12.31; destination host is: vm-33:25000; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1414) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getListing(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:519) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1767) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1750) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:691) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:753) at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:749) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:749) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:68) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:677) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1641) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:640) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:724) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462) at org.apache.hadoop.ipc.Client.call(Client.java:1381) ... 21 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at
[jira] [Commented] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[
https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574202#comment-14574202
]
Devaraj K commented on MAPREDUCE-6350:
--
Thanks [~l201514] for the patch.
Patch mostly looks good except these nits.
1. Can you move the new configuration mapreduce.jobhistory.jobname.limit and
the default value to JHAdminConfig.java.
{code:xml}
+ private static final int JOB_NAME_TRIM_LENGTH = 50;
{code}
{code:xml}
+int jobNameLimit =
+getConfig().getInt(mapreduce.jobhistory.jobname.limit,
JOB_NAME_TRIM_LENGTH);
{code}
2. In FileNameIndexUtils.java, can you remove the JOB_NAME_TRIM_LENGTH
declaration and access it from JHAdminConfig.java.
3. Please add tests for covering this change.
JobHistory doesn't support fully-functional search
--
Key: MAPREDUCE-6350
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch,
YARN-1614.v2.patch, YARN-1614.v3.patch
job history server will only output the first 50 characters of the job names
in webUI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[ https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6350: - Target Version/s: 2.8.0 Status: Open (was: Patch Available) JobHistory doesn't support fully-functional search -- Key: MAPREDUCE-6350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch, YARN-1614.v2.patch, YARN-1614.v3.patch job history server will only output the first 50 characters of the job names in webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6377: - Hadoop Flags: Reviewed I verified with the patch, it fixes the issue. +1, patch looks good to me. JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6377: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~zxu] for the patch and [~bibinchundatt] for reporting the issue. Committed to trunk and branch-2. JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574568#comment-14574568 ] Hudson commented on MAPREDUCE-6377: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2147 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2147/]) MAPREDUCE-6377. JHS sorting on state column not working in webUi. (devaraj: rev 790a861766dcd60212d83f444f2f96d3acf20a94) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575010#comment-14575010
]
zhihai xu commented on MAPREDUCE-6384:
--
Hi [~lichangleo], thanks for working on this issue. this looks like a good
improvement to show more diagnostic information about fetch failure.
In your patch, I think you want to get the host name for the reduce task.
But it looks like you use the host name for Application Master as
reduceHostname in the following code.
{code}
hostname = InetAddress.getLocalHost().getHostName();
{code}
I think the local host is the AM host not reduce task host, it will be correct
only when both AM and reduce tasks are running on the same host.
I think we should use {{this.container.getNodeId().getHost()}} as
reduceHostname, which is the same as containerHostName.
Also it looks like the following line is more than 80 characters .
{code}
Arrays.asList(new TaskAttemptId[] {mapAttempt.getID()}),
sudoHost));
{code}
add the last reporting reducer host info and attempt id on the map error
message due to too many fetch failure
--
Key: MAPREDUCE-6384
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574755#comment-14574755
]
Karthik Kambatla commented on MAPREDUCE-6387:
-
Checking this in.
Serialize the recently added Task#encryptedSpillKey field at the end
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-6387:
Target Version/s: 2.7.1
Affects Version/s: 2.7.0
Serialize the recently added Task#encryptedSpillKey field at the end
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574711#comment-14574711 ] Eric Payne commented on MAPREDUCE-6174: --- Thank you, [~jira.shegalov]! Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. --- Key: MAPREDUCE-6174 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.0.0, 2.6.0 Reporter: Eric Payne Assignee: Eric Payne Labels: BB2015-05-RFC Fix For: 2.8.0 Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.006.patch, MAPREDUCE-6174.007.patch, MAPREDUCE-6174.v1.txt Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing similar things with regards to IFile streams. In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are different from 3rd-party implementations, this JIRA will make them subclass a common class (see https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-6387:
Summary: Serialize the recently added Task#encryptedSpillKey field at the
end (was: Fix Task object to allow Jobs not submitted via distributed cache to
survive rolling upgrade)
Serialize the recently added Task#encryptedSpillKey field at the end
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-6387:
Resolution: Fixed
Fix Version/s: 2.7.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for working on this, Arun.
Just committed this to trunk, branch-2, and branch-2.7.
Serialize the recently added Task#encryptedSpillKey field at the end
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Fix For: 2.7.1
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6387) Serialize the recently added Task#encryptedSpillKey field at the end
[
https://issues.apache.org/jira/browse/MAPREDUCE-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574776#comment-14574776
]
Hudson commented on MAPREDUCE-6387:
---
FAILURE: Integrated in Hadoop-trunk-Commit #7975 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7975/])
MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at
the end. (Arun Suresh via kasha) (kasha: rev
6786daab33792c9c426adf0fc6b361f193d356a8)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
Serialize the recently added Task#encryptedSpillKey field at the end
Key: MAPREDUCE-6387
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6387
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
Fix For: 2.7.1
Attachments: MAPREDUCE-6387.1.patch
There was a recent addition of an {{encryptedSpillKey}} to the Task object.
And when serialized, this field was written out somewhere in the middle. This
caused deployments that do not use DistributedCache to push job jars before
running the job to fail rolling upgrade.
Although deploying via Distributed Cache is the recommended method, there
might still be deployments that use the node local classpath to pick up the
MR framework classes (eg. for efficiency purposes, since this does not
require the jar being copied to hdfs and then to all the nodes)
Ensuring that it is the last field written and read when the Task object is
serialized would alleviate this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6383) Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6383: - Hadoop Flags: Reviewed +1, patch looks good to me, will commit it shortly. Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails - Key: MAPREDUCE-6383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6383 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Affects Versions: 2.7.0 Reporter: Harsh J Assignee: Harsh J Attachments: MAPREDUCE-6383.patch Currently it tries to read the standard result path and logs a failure a second time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6383) Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6383: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~qwertymaniac]. Committed to trunk and branch-2. Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails - Key: MAPREDUCE-6383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6383 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Affects Versions: 2.7.0 Reporter: Harsh J Assignee: Harsh J Fix For: 2.8.0 Attachments: MAPREDUCE-6383.patch Currently it tries to read the standard result path and logs a failure a second time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6383) Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574741#comment-14574741 ] Hudson commented on MAPREDUCE-6383: --- FAILURE: Integrated in Hadoop-trunk-Commit #7974 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7974/]) MAPREDUCE-6383. Pi job (QuasiMonteCarlo) should not try to read the (devaraj: rev 09b8241e6e369ee00241efcc2e048e710dbcccd1) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java Pi job (QuasiMonteCarlo) should not try to read the results file if its job fails - Key: MAPREDUCE-6383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6383 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Affects Versions: 2.7.0 Reporter: Harsh J Assignee: Harsh J Fix For: 2.8.0 Attachments: MAPREDUCE-6383.patch Currently it tries to read the standard result path and logs a failure a second time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574653#comment-14574653 ] Hudson commented on MAPREDUCE-6377: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #217 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/217/]) MAPREDUCE-6377. JHS sorting on state column not working in webUi. (devaraj: rev 790a861766dcd60212d83f444f2f96d3acf20a94) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java * hadoop-mapreduce-project/CHANGES.txt JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[ https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6350: --- Attachment: YARN-1614.v5.patch JobHistory doesn't support fully-functional search -- Key: MAPREDUCE-6350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch, YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch, YARN-1614.v5.patch job history server will only output the first 50 characters of the job names in webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575067#comment-14575067 ] zhihai xu commented on MAPREDUCE-6377: -- thanks [~devaraj.k] for reviewing and committing the patch, thanks [~bibinchundatt] for reporting the issue. JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[ https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6350: --- Attachment: YARN-1614.v4.patch JobHistory doesn't support fully-functional search -- Key: MAPREDUCE-6350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch, YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch job history server will only output the first 50 characters of the job names in webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[ https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-6350: --- Status: Patch Available (was: Open) JobHistory doesn't support fully-functional search -- Key: MAPREDUCE-6350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch, YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch job history server will only output the first 50 characters of the job names in webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6384: Attachment: MAPREDUCE-6384.5.patch Hi [~zxu], thanks for review and good catch! I just updated my patch according to your suggestion. add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure -- Key: MAPREDUCE-6384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch, MAPREDUCE-6384.5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[
https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575305#comment-14575305
]
Hadoop QA commented on MAPREDUCE-6350:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 57s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 3s | The applied patch generated 1
new checkstyle issues (total was 49, now 50). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 18s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 23s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| | | 50m 43s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738051/YARN-1614.v5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / bc11e15 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5777/console |
This message was automatically generated.
JobHistory doesn't support fully-functional search
--
Key: MAPREDUCE-6350
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch,
YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch, YARN-1614.v5.patch
job history server will only output the first 50 characters of the job names
in webUI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6384: - Issue Type: Improvement (was: Bug) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure -- Key: MAPREDUCE-6384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch, MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sahitya pavurala reassigned MAPREDUCE-3418: --- Assignee: sahitya pavurala (was: Chen He) If map output is not found, shuffle runs in tight loop -- Key: MAPREDUCE-3418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 2.3.0 Reporter: John George Assignee: sahitya pavurala Sharad Agarwal bumped into this while simulating fetch failures. Removed the map output directory. Shuffle runs in tight loop throwing : 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[
https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575186#comment-14575186
]
Hadoop QA commented on MAPREDUCE-6350:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 46s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 45s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 3s | The applied patch generated 4
new checkstyle issues (total was 49, now 53). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 19s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 30s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| | | 50m 16s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738021/YARN-1614.v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3e000a9 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5776/console |
This message was automatically generated.
JobHistory doesn't support fully-functional search
--
Key: MAPREDUCE-6350
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: Siqi Li
Assignee: Siqi Li
Priority: Critical
Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch,
YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch
job history server will only output the first 50 characters of the job names
in webUI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) ShuffleHandler should be able to log shuffle connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6354: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks, Chang! I committed this to trunk and branch-2. ShuffleHandler should be able to log shuffle connections Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) ShuffleHandler should be able to log shuffle connections
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575371#comment-14575371 ] Hudson commented on MAPREDUCE-6354: --- SUCCESS: Integrated in Hadoop-trunk-Commit #7981 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7981/]) MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li (jlowe: rev b3ffa870034d06608a1946e2d9ce7dbd535a2c53) * hadoop-mapreduce-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/conf/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java ShuffleHandler should be able to log shuffle connections Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Chang Li Assignee: Chang Li Fix For: 2.8.0 Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.7.patch, MAPREDUCE-6354.8.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6388) Remove deprecation warnings from JobHistoryServer classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6388: -- Attachment: MAPREDUCE-6388.001.patch - Replace a lot of direct member variable access with appropriate get/set method - Fix some other calls to deprecated methods - Fix a few column width violations Remove deprecation warnings from JobHistoryServer classes - Key: MAPREDUCE-6388 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6388 Project: Hadoop Map/Reduce Issue Type: Task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6388.001.patch There are a ton of deprecation warnings in the JobHistoryServer classes. This is affecting some modifications I'm making since a single line move shifts all the deprecation warnings. I'd like to get these fixed to prevent minor changes from generating a ton of warnings in test-patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6388) Remove deprecation warnings from JobHistoryServer classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6388: -- Status: Patch Available (was: Open) Submit for testing. Remove deprecation warnings from JobHistoryServer classes - Key: MAPREDUCE-6388 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6388 Project: Hadoop Map/Reduce Issue Type: Task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6388.001.patch There are a ton of deprecation warnings in the JobHistoryServer classes. This is affecting some modifications I'm making since a single line move shifts all the deprecation warnings. I'd like to get these fixed to prevent minor changes from generating a ton of warnings in test-patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6350) JobHistory doesn't support fully-functional search
[ https://issues.apache.org/jira/browse/MAPREDUCE-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575340#comment-14575340 ] Siqi Li commented on MAPREDUCE-6350: Thanks [~devaraj.k] for your feedback, I have uploaded patch v5 that addressed those nits and added a test case. JobHistory doesn't support fully-functional search -- Key: MAPREDUCE-6350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6350 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: Siqi Li Assignee: Siqi Li Priority: Critical Attachments: MAPREDUCE-6350.v1.patch, YARN-1614.v1.patch, YARN-1614.v2.patch, YARN-1614.v3.patch, YARN-1614.v4.patch, YARN-1614.v5.patch job history server will only output the first 50 characters of the job names in webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6384) add the last reporting reducer host info and attempt id on the map error message due to too many fetch failure
[
https://issues.apache.org/jira/browse/MAPREDUCE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575345#comment-14575345
]
Hadoop QA commented on MAPREDUCE-6384:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 36s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 32s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 34s | The applied patch generated 2
new checkstyle issues (total was 357, now 356). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 56s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 47m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738060/MAPREDUCE-6384.5.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / bc11e15 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5778/console |
This message was automatically generated.
add the last reporting reducer host info and attempt id on the map error
message due to too many fetch failure
--
Key: MAPREDUCE-6384
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6384
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6384.2.patch, MAPREDUCE-6384.3.patch,
MAPREDUCE-6384.4.2.patch, MAPREDUCE-6384.4.patch, MAPREDUCE-6384.5.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6388) Remove deprecation warnings from JobHistoryServer classes
Ray Chiang created MAPREDUCE-6388: - Summary: Remove deprecation warnings from JobHistoryServer classes Key: MAPREDUCE-6388 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6388 Project: Hadoop Map/Reduce Issue Type: Task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor There are a ton of deprecation warnings in the JobHistoryServer classes. This is affecting some modifications I'm making since a single line move shifts all the deprecation warnings. I'd like to get these fixed to prevent minor changes from generating a ton of warnings in test-patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574601#comment-14574601 ] Hudson commented on MAPREDUCE-6377: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #208 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/208/]) MAPREDUCE-6377. JHS sorting on state column not working in webUi. (devaraj: rev 790a861766dcd60212d83f444f2f96d3acf20a94) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574625#comment-14574625 ] Hudson commented on MAPREDUCE-6377: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2165 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2165/]) MAPREDUCE-6377. JHS sorting on state column not working in webUi. (devaraj: rev 790a861766dcd60212d83f444f2f96d3acf20a94) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, state_sorted1.pdf, state_sorted2.pdf Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
