[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702056#comment-13702056 ] Avner BenHanoch commented on MAPREDUCE-4049: Hi Arun, and Alejandro, and anyone else, I would like to revive the branch-1 patch in this JIRA issue and submit it to the updated branch-1. *This patch was already reviewed by Alejandro and got +1 in 05/Feb/13* Please let me know what else is needed for closing this story. Thanks, Avner plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: performance, task, tasktracker Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 Reporter: Avner BenHanoch Assignee: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 2.0.3-alpha Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) # I am providing link for downloading UDA - Mellanox's open source plugin that implements generic shuffle service using RDMA and levitated merge. Note: At this phase, the code is in C++ through JNI and you should consider it as beta only. Still, it can serve anyone that wants to implement or contribute to levitated merge. (Please be advised that levitated merge is mostly suit in very fast networks) - [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5187) Create mapreduce command scripts on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-5187: - Target Version/s: 3.0.0, 2.1.0-beta Affects Version/s: 2.1.0-beta Hadoop Flags: Reviewed +1 from me too. I'll commit this later today. Create mapreduce command scripts on Windows --- Key: MAPREDUCE-5187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5187 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Attachments: MAPREDUCE-5187-trunk.2.patch, MAPREDUCE-5187-trunk.patch We don't have mapreduce command scripts, e.g. mapred.cmd, on Windows in trunk code base right now. As a result, some import functionality like Job history server is not available. This JIRA is created to track this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5366) TestMRAsyncDiskService fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-5366:
-
Component/s: test
Target Version/s: 3.0.0, 2.1.0-beta
Hadoop Flags: Reviewed
+1 for the patch. I verified tests on Mac and Windows. I'll commit this.
TestMRAsyncDiskService fails on Windows
---
Key: MAPREDUCE-5366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Attachments: MAPREDUCE-5366-trunk.patch
The unit test fails on Windows because mismatching of {{File.seperator}} and
{{Path}} object name. In general, we should only use {{Path.SEPARATOR}} when
dealing with {{Path}} objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5366) TestMRAsyncDiskService fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702138#comment-13702138
]
Hudson commented on MAPREDUCE-5366:
---
Integrated in Hadoop-trunk-Commit #4048 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4048/])
MAPREDUCE-5366. TestMRAsyncDiskService fails on Windows. Contributed by
Chuan Liu. (Revision 1500842)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500842
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestMRAsyncDiskService.java
TestMRAsyncDiskService fails on Windows
---
Key: MAPREDUCE-5366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Attachments: MAPREDUCE-5366-trunk.patch
The unit test fails on Windows because mismatching of {{File.seperator}} and
{{Path}} object name. In general, we should only use {{Path.SEPARATOR}} when
dealing with {{Path}} objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5366) TestMRAsyncDiskService fails on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-5366:
-
Resolution: Fixed
Fix Version/s: 2.1.0-beta
3.0.0
Status: Resolved (was: Patch Available)
I committed this to trunk, branch-2, branch-2.1-beta, and branch-2.1.0-beta.
Thank you, Chuan.
TestMRAsyncDiskService fails on Windows
---
Key: MAPREDUCE-5366
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5366
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5366-trunk.patch
The unit test fails on Windows because mismatching of {{File.seperator}} and
{{Path}} object name. In general, we should only use {{Path.SEPARATOR}} when
dealing with {{Path}} objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5371) TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users
[ https://issues.apache.org/jira/browse/MAPREDUCE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-5371: - Component/s: test Target Version/s: 1-win Hadoop Flags: Reviewed TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users --- Key: MAPREDUCE-5371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5371 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Fix For: 1-win Attachments: MAPREDUCE-5371.patch The error message was: Error Message expected:[sijenkins-vm2]jenkins but was:[]jenkins Stacktrace at org.apache.hadoop.security.TestProxyUserFromEnv.testProxyUserFromEnvironment(TestProxyUserFromEnv.java:45) The root cause of this failure is the domain used on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5371) TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users
[ https://issues.apache.org/jira/browse/MAPREDUCE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved MAPREDUCE-5371. -- Resolution: Fixed +1 for the patch. I committed this to branch-1-win. Thank you, Xi. TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users --- Key: MAPREDUCE-5371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5371 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Fix For: 1-win Attachments: MAPREDUCE-5371.patch The error message was: Error Message expected:[sijenkins-vm2]jenkins but was:[]jenkins Stacktrace at org.apache.hadoop.security.TestProxyUserFromEnv.testProxyUserFromEnvironment(TestProxyUserFromEnv.java:45) The root cause of this failure is the domain used on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5371) TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users
[ https://issues.apache.org/jira/browse/MAPREDUCE-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702176#comment-13702176 ] Xi Fang commented on MAPREDUCE-5371: Thanks Chris! TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users --- Key: MAPREDUCE-5371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5371 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1-win Environment: Windows Reporter: Xi Fang Assignee: Xi Fang Priority: Minor Fix For: 1-win Attachments: MAPREDUCE-5371.patch The error message was: Error Message expected:[sijenkins-vm2]jenkins but was:[]jenkins Stacktrace at org.apache.hadoop.security.TestProxyUserFromEnv.testProxyUserFromEnvironment(TestProxyUserFromEnv.java:45) The root cause of this failure is the domain used on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5278) Distributed cache is broken when JT staging dir is not on the default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702203#comment-13702203
]
Chris Nauroth commented on MAPREDUCE-5278:
--
{code}
public void testJTStagingOnNondefaultFS() throws Exception {
MiniDFSCluster dfs = null;
MiniMRCluster mr = null;
try {
dfs = new MiniDFSCluster(conf, 1, true, null);
FileSystem fileSys = dfs.getFileSystem();
mr = new MiniMRCluster(1, fileSys.getUri().toString(), 1);
runWithConfJTStagingOnNondefaultFS(mr.createJobConf());
} finally {
mr.shutdown();
dfs.shutdown();
}
}
{code}
In the finally block, could you please check that mr and dfs are not null
before calling shutdown on each one? The constructors can throw exceptions, so
it's possible that they weren't initialized.
Aside from that, it appears that all of the feedback from Bikas has been
addressed. I think we'll be ready to commit once this last part has been
fixed. Thanks, Xi!
Distributed cache is broken when JT staging dir is not on the default FS
Key: MAPREDUCE-5278
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5278
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache
Affects Versions: 1-win
Environment: Windows
Reporter: Xi Fang
Assignee: Xi Fang
Fix For: 1-win
Attachments: MAPREDUCE-5278.2.patch, MAPREDUCE-5278.3.patch,
MAPREDUCE-5278.4.patch, MAPREDUCE-5278.patch
Today, the JobTracker staging dir (mapreduce.jobtracker.staging.root.dir) is
set to point to HDFS, even though other file systems (e.g. Amazon S3 file
system and Windows ASV file system) are the default file systems.
For ASV, this config was chosen and there are a few reasons why:
1. To prevent leak of the storage account credentials to the user's storage
account;
2. It uses HDFS for the transient job files what is good for two reasons – a)
it does not flood the user's storage account with irrelevant data/files b) it
leverages HDFS locality for small files
However, this approach conflicts with how distributed cache caching works,
completely negating the feature's functionality.
When files are added to the distributed cache (thru files/achieves/libjars
hadoop generic options), they are copied to the job tracker staging dir only
if they reside on a file system different that the jobtracker's. Later on,
this path is used as a key to cache the files locally on the tasktracker's
machine, and avoid localization (download/unzip) of the distributed cache
files if they are already localized.
In this configuration the caching is completely disabled and we always end up
copying dist cache files to the job tracker's staging dir first and
localizing them on the task tracker machine second.
This is especially not good for Oozie scenarios as Oozie uses dist cache to
populate Hive/Pig jars throughout the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5187) Create mapreduce command scripts on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-5187: - Resolution: Fixed Fix Version/s: 2.1.0-beta 3.0.0 Status: Resolved (was: Patch Available) I committed this to trunk, branch-2, branch-2.1-beta, and branch-2.1.0-beta. Thank you for the contribution, Chuan. Create mapreduce command scripts on Windows --- Key: MAPREDUCE-5187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5187 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 3.0.0, 2.1.0-beta Attachments: MAPREDUCE-5187-trunk.2.patch, MAPREDUCE-5187-trunk.patch We don't have mapreduce command scripts, e.g. mapred.cmd, on Windows in trunk code base right now. As a result, some import functionality like Job history server is not available. This JIRA is created to track this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5187) Create mapreduce command scripts on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702390#comment-13702390 ] Hudson commented on MAPREDUCE-5187: --- Integrated in Hadoop-trunk-Commit #4050 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4050/]) MAPREDUCE-5187. Create mapreduce command scripts on Windows. Contributed by Chuan Liu. (Revision 1500929) Result = SUCCESS cnauroth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500929 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.cmd * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd * /hadoop/common/trunk/hadoop-mapreduce-project/conf/mapred-env.cmd Create mapreduce command scripts on Windows --- Key: MAPREDUCE-5187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5187 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0, 2.1.0-beta Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 3.0.0, 2.1.0-beta Attachments: MAPREDUCE-5187-trunk.2.patch, MAPREDUCE-5187-trunk.patch We don't have mapreduce command scripts, e.g. mapred.cmd, on Windows in trunk code base right now. As a result, some import functionality like Job history server is not available. This JIRA is created to track this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5376) The display of the counter of Map column and Reduce column uses displayName in jobdetails.jsp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita reassigned MAPREDUCE-5376: - Assignee: Shinichi Yamashita The display of the counter of Map column and Reduce column uses displayName in jobdetails.jsp - Key: MAPREDUCE-5376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5376 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, mrv1 Affects Versions: 1.2.0 Reporter: Shinichi Yamashita Assignee: Shinichi Yamashita Attachments: MAPREDUCE-5376.patch In jobdetails.jsp, the The acquisition of the counter of M/R column uses displayName in Counters.Group.getCounter(). Therefore, the counter where is different in realname and displayname cannot get a value. It uses displayName to get a value by Map column and Reduce column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5377) HistoryViewer.java sets wrong jobId
[ https://issues.apache.org/jira/browse/MAPREDUCE-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita reassigned MAPREDUCE-5377: - Assignee: Shinichi Yamashita HistoryViewer.java sets wrong jobId --- Key: MAPREDUCE-5377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5377 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Shinichi Yamashita Assignee: Shinichi Yamashita Attachments: MAPREDUCE-5377.patch JobID output by hadoop job -history is wrong. This problem can solve by fixing setting of jobId in HistoryViewer(). In addition, it should fix the information of JobTracker at HistoryViewr. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5364) Deadlock between RenewalTimerTask methods cancel() and run()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5364:
Attachment: mr-5364-addendum-2.patch
Deadlock between RenewalTimerTask methods cancel() and run()
Key: MAPREDUCE-5364
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5364
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 1.2.1
Attachments: mr-5364-1.patch, mr-5364-addendum-1.patch,
mr-5364-addendum-2.patch
MAPREDUCE-4860 introduced a local variable {{cancelled}} in
{{RenewalTimerTask}} to fix the race where {{DelegationTokenRenewal}}
attempts to renew a token even after the job is removed. However, the patch
also makes {{run()}} and {{cancel()}} synchronized methods leading to a
potential deadlock against {{run()}}'s catch-block (error-path).
The deadlock stacks below:
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.cancel()
@bci=0, line=240 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeDelegationTokenRenewalForJob(org.apache.hadoop.mapreduce.JobID)
@bci=109, line=319 (Interpreted frame)
{noformat}
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeFailedDelegationToken(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=62, line=297 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.access$300(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=1, line=47 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run()
@bci=148, line=234 (Interpreted frame)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5364) Deadlock between RenewalTimerTask methods cancel() and run()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702489#comment-13702489
]
Karthik Kambatla commented on MAPREDUCE-5364:
-
bq. A cancelled flag could be used on the DelegationTokenToRenew structure
itself. Set intent to cancel before attempting to cancel the timer task, and
check this during renewal and before queuing another renewal.
On second thought, I don't think we should a cancelled flag to
DelegationTokenToRenew to address a synchronization in RenewalTimerTask.
Thinking more about this, the cleanest approach seemed to be:
# Serialize timer cancellation and token renewal - via synchronization
# On successful token renewal, call {{setTimerForTokenRenewal}}
Uploaded patch (addendum-2) to implement this. Also, moved the token renewal
code from RenewalTimerTask#run to DelegationTokenToRenew#renew for clarity.
[~sseth], can you take a look at the latest patch.
Deadlock between RenewalTimerTask methods cancel() and run()
Key: MAPREDUCE-5364
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5364
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 1.2.1
Attachments: mr-5364-1.patch, mr-5364-addendum-1.patch,
mr-5364-addendum-2.patch
MAPREDUCE-4860 introduced a local variable {{cancelled}} in
{{RenewalTimerTask}} to fix the race where {{DelegationTokenRenewal}}
attempts to renew a token even after the job is removed. However, the patch
also makes {{run()}} and {{cancel()}} synchronized methods leading to a
potential deadlock against {{run()}}'s catch-block (error-path).
The deadlock stacks below:
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.cancel()
@bci=0, line=240 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeDelegationTokenRenewalForJob(org.apache.hadoop.mapreduce.JobID)
@bci=109, line=319 (Interpreted frame)
{noformat}
{noformat}
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.removeFailedDelegationToken(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=62, line=297 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal.access$300(org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$DelegationTokenToRenew)
@bci=1, line=47 (Interpreted frame)
-
org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run()
@bci=148, line=234 (Interpreted frame)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5325) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5325: - Attachment: MR-5325.2.patch ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes - Key: MAPREDUCE-5325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5325 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: MR-5325.1.patch, MR-5325.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5378) Make the QOP for application master to be derived based on
Benoy Antony created MAPREDUCE-5378: --- Summary: Make the QOP for application master to be derived based on Key: MAPREDUCE-5378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5378 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Currently ApplicationMaster's QOP(quality of protection) is derived from the client's config. If the client uses privacy, all the communication done by the application master will be set to privacy. As part of the feature to support multiple QOP (HADOOP -9709), the application master is modified so that application master can support a different QOPs for its communication with client vs its communication with other hadoop components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5378) Make the QOP for application master to be derived based on
[ https://issues.apache.org/jira/browse/MAPREDUCE-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-5378: Attachment: mr-5378.patch I'll add the testcases once the approach is reviewed. Make the QOP for application master to be derived based on --- Key: MAPREDUCE-5378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5378 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: mr-5378.patch Currently ApplicationMaster's QOP(quality of protection) is derived from the client's config. If the client uses privacy, all the communication done by the application master will be set to privacy. As part of the feature to support multiple QOP (HADOOP -9709), the application master is modified so that application master can support a different QOPs for its communication with client vs its communication with other hadoop components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5378) Enable ApplicationMaster to support different QOP for client and server communications
[ https://issues.apache.org/jira/browse/MAPREDUCE-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-5378: Summary: Enable ApplicationMaster to support different QOP for client and server communications (was: Make the QOP for application master to be derived based on ) Enable ApplicationMaster to support different QOP for client and server communications -- Key: MAPREDUCE-5378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5378 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: mr-5378.patch Currently ApplicationMaster's QOP(quality of protection) is derived from the client's config. If the client uses privacy, all the communication done by the application master will be set to privacy. As part of the feature to support multiple QOP (HADOOP -9709), the application master is modified so that application master can support a different QOPs for its communication with client vs its communication with other hadoop components. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5325) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5325: - Attachment: MR-5325.3.patch ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes - Key: MAPREDUCE-5325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5325 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: MR-5325.1.patch, MR-5325.2.patch, MR-5325.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5377) HistoryViewer.java sets wrong jobId
[ https://issues.apache.org/jira/browse/MAPREDUCE-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated MAPREDUCE-5377: -- Status: Patch Available (was: Open) HistoryViewer.java sets wrong jobId --- Key: MAPREDUCE-5377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5377 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Shinichi Yamashita Assignee: Shinichi Yamashita Attachments: MAPREDUCE-5377.patch JobID output by hadoop job -history is wrong. This problem can solve by fixing setting of jobId in HistoryViewer(). In addition, it should fix the information of JobTracker at HistoryViewr. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5377) HistoryViewer.java sets wrong jobId
[ https://issues.apache.org/jira/browse/MAPREDUCE-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702790#comment-13702790 ] Shinichi Yamashita commented on MAPREDUCE-5377: --- I attach a patch file. HistoryViewer.java sets wrong jobId --- Key: MAPREDUCE-5377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5377 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.0 Reporter: Shinichi Yamashita Assignee: Shinichi Yamashita Attachments: MAPREDUCE-5377.patch JobID output by hadoop job -history is wrong. This problem can solve by fixing setting of jobId in HistoryViewer(). In addition, it should fix the information of JobTracker at HistoryViewr. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5377) HistoryViewer.java sets wrong jobId
[
https://issues.apache.org/jira/browse/MAPREDUCE-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702814#comment-13702814
]
Hadoop QA commented on MAPREDUCE-5377:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591055/MAPREDUCE-5377.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3840//console
This message is automatically generated.
HistoryViewer.java sets wrong jobId
---
Key: MAPREDUCE-5377
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5377
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.2.0
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Attachments: MAPREDUCE-5377.patch
JobID output by hadoop job -history is wrong. This problem can solve by
fixing setting of jobId in HistoryViewer().
In addition, it should fix the information of JobTracker at HistoryViewr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5346) The example provided in the javadoc for Job is outdated.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702924#comment-13702924 ] Tsuyoshi OZAWA commented on MAPREDUCE-5346: --- +1 The example provided in the javadoc for Job is outdated. Key: MAPREDUCE-5346 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5346 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.1.0-beta, 2.0.5-alpha Reporter: Pranay Varma Assignee: Akira AJISAKA Labels: newbie Attachments: MAPREDUCE-5346-1.patch This is a portion of the job example: Here is an example on how to submit a job: // Create a new Job Job job = new Job(new Configuration()); This way of creating a job is deprecated. The user is supposed to use Job.getInstance. The example should be changed to reflect the new paradigm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5346) The example provided in the javadoc for Job is outdated.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5346: -- Hadoop Flags: Reviewed The example provided in the javadoc for Job is outdated. Key: MAPREDUCE-5346 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5346 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.1.0-beta, 2.0.5-alpha Reporter: Pranay Varma Assignee: Akira AJISAKA Labels: newbie Attachments: MAPREDUCE-5346-1.patch This is a portion of the job example: Here is an example on how to submit a job: // Create a new Job Job job = new Job(new Configuration()); This way of creating a job is deprecated. The user is supposed to use Job.getInstance. The example should be changed to reflect the new paradigm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5294) Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5294: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Marked as a duplicate of MAPREDUCE-5221. Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer --- Key: MAPREDUCE-5294 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5294 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-5294.1.patch, MAPREDUCE-5294.2.patch, MAPREDUCE-5294.3.patch Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5295) ShuffleConsumerPlugin.Context should remove unused combiner-related methods and fields
[ https://issues.apache.org/jira/browse/MAPREDUCE-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA resolved MAPREDUCE-5295. --- Resolution: Duplicate Marked as a duplicate of MAPREDUCE-5221. ShuffleConsumerPlugin.Context should remove unused combiner-related methods and fields -- Key: MAPREDUCE-5295 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5295 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: trunk, 2.1.0-beta, 2.0.5-alpha Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: MAPREDUCE-5295.1.patch, MAPREDUCE-5295.2.patch, MAPREDUCE-5295.3.patch ShuffleConsumerPlugin.Context only supports org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce side Combiner is not used when using the new API, and just ignored. Please see MAPREDUCE-5221 for more detail. (Update Jun. 7th, 2013) The latest version of MergeManagerImpl in MAPREDUCE-5294 never calls ShuffleConsumerPlugin.Context#getCombinerClass() as [~masokan] pointed out. Because of this, combinerClass field and getCombinerClass() method should be removed from ShuffleConsumerPlugin.Context. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
