[jira] [Commented] (MAPREDUCE-4893) MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567529#comment-13567529 ] Hudson commented on MAPREDUCE-4893: --- Integrated in Hadoop-Yarn-trunk #113 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/113/]) MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of containers to get maximum locality. Contributed by Bikas Saha. (Revision 1440749) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1440749 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality --- Key: MAPREDUCE-4893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4893 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4893.1.patch, MAPREDUCE-4893.2.patch, MAPREDUCE-4893.3.patch Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map-container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4893) MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567603#comment-13567603 ] Hudson commented on MAPREDUCE-4893: --- Integrated in Hadoop-Hdfs-trunk #1302 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1302/]) MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of containers to get maximum locality. Contributed by Bikas Saha. (Revision 1440749) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1440749 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality --- Key: MAPREDUCE-4893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4893 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4893.1.patch, MAPREDUCE-4893.2.patch, MAPREDUCE-4893.3.patch Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map-container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4893) MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567652#comment-13567652 ] Hudson commented on MAPREDUCE-4893: --- Integrated in Hadoop-Mapreduce-trunk #1330 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1330/]) MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of containers to get maximum locality. Contributed by Bikas Saha. (Revision 1440749) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1440749 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality --- Key: MAPREDUCE-4893 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4893 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4893.1.patch, MAPREDUCE-4893.2.patch, MAPREDUCE-4893.3.patch Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map-container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567748#comment-13567748 ] Avner BenHanoch commented on MAPREDUCE-4049: Hi Alejadro Arun, Thank you for your review and all your comments. I appreciate your help and responsiveness with my issue. I would like to say a few comments/answers before the patch is concluded: 1. *_Alejandro_* - _getJobConf(JobID)_ is needed for any ShuffleProvider. The provider needs it for determining _username_ and _runAsUsername_. _username_ is needed for determining the location in disk of the MOF and Index files. _runAsUsername_ is needed for reading the above files with the right privileges. 2. *_Alejandro_* – The answer for your question about the tests is - YES. I did run all smoke commit tests successfully. 3. *_Arun_* - I have no problem with your request for not passing the entire ReduceTask. I am only a bit worried about initing ShuffleConsumerPlugin with arguments such as _getPartition()_ and _getJobTokenSecret()_. The reason is that at least theoretically it is possible to change _partition/jobTokenSecret_ after the shuffleConsumerPlugin was initiated. Hence, I need your approval for that. Additionally, please notice that in hadoop-trunk we do pass the entire ReduceTask to the ShuffleConsumerPlugin. (Also, in hadoop-1 we always passed ReduceTask. I think that with the last patch it is highlighted because we made ReduceCopier a static class which required specifying explicitly reduceTask.XXX in about 75 different places). *_Bottom line, Arun, please let me know if you are still worried about passing the entire ReduceTask to the shuffle plugin._* thank you, Avner plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: performance, task, tasktracker Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 Reporter: Avner BenHanoch Assignee: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 2.0.3-alpha Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) # I am providing link for downloading UDA - Mellanox's open source plugin that implements generic shuffle service using RDMA and levitated merge. Note: At this phase, the code is in C++ through JNI and you should consider it as beta only. Still, it can serve anyone that wants to implement or contribute to levitated merge. (Please be advised that levitated merge is mostly suit in very fast networks) - [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4491) Encryption and Key Protection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1356#comment-1356 ] Benoy Antony commented on MAPREDUCE-4491: - Yes, That makes sense. Encryption and Key Protection - Key: MAPREDUCE-4491 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4491 Project: Hadoop Map/Reduce Issue Type: New Feature Components: documentation, security, task-controller, tasktracker Reporter: Benoy Antony Assignee: Benoy Antony Attachments: crypto_abstractions.zip, Hadoop_Encryption.pdf, Hadoop_Encryption.pdf When dealing with sensitive data, it is required to keep the data encrypted wherever it is stored. Common use case is to pull encrypted data out of a datasource and store in HDFS for analysis. The keys are stored in an external keystore. The feature adds a customizable framework to integrate different types of keystores, support for Java KeyStore, read keys from keystores, and transport keys from JobClient to Tasks. The feature adds PGP encryption as a codec and additional utilities to perform encryption related steps. The design document is attached. It explains the requirement, design and use cases. Kindly review and comment. Collaboration is very much welcome. I have a tested patch for this for 1.1 and will upload it soon as an initial work for further refinement. Update: The patches are uploaded to subtasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-4969) RawLocalFileSystem.delete unexpected behavior on Windows while running Mapreduce tests with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas moved HADOOP-9176 to MAPREDUCE-4969: Component/s: (was: test) test Fix Version/s: (was: 1.2.0) (was: 3.0.0) 3.0.0 1.2.0 Affects Version/s: (was: trunk-win) (was: 1.2.0) (was: 1-win) (was: 3.0.0) trunk-win 1-win 3.0.0 1.2.0 Key: MAPREDUCE-4969 (was: HADOOP-9176) Project: Hadoop Map/Reduce (was: Hadoop Common) RawLocalFileSystem.delete unexpected behavior on Windows while running Mapreduce tests with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 3.0.0 Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4969: --- Summary: TestKeyValueTextInputFormat test fails with Open JDK 7 (was: RawLocalFileSystem.delete unexpected behavior on Windows while running Mapreduce tests with Open JDK 7) TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 3.0.0 Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567822#comment-13567822 ] Suresh Srinivas commented on MAPREDUCE-4969: +1 for the branch-1 patch as well. TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 3.0.0 Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567840#comment-13567840 ] Hudson commented on MAPREDUCE-4969: --- Integrated in Hadoop-trunk-Commit #3304 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3304/]) MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7. Contributed by Arpit Agarwal. (Revision 1441069) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1441069 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestKeyValueTextInputFormat.java TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 3.0.0 Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4969: --- Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.0.3-alpha Status: Resolved (was: Patch Available) I committed this patch to trunk, branch-2, branch-1 and branch-1-win. Phew... Thank you Arpit! TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 2.0.3-alpha Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4969: --- Hadoop Flags: Reviewed TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 2.0.3-alpha Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567847#comment-13567847 ] Arpit Agarwal commented on MAPREDUCE-4969: -- Thanks, Suresh and thanks for fixing the JIRA fields. TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 2.0.3-alpha Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[
https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567881#comment-13567881
]
Hadoop QA commented on MAPREDUCE-4969:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567287/HADOOP-9176.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3294//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3294//console
This message is automatically generated.
TestKeyValueTextInputFormat test fails with Open JDK 7
--
Key: MAPREDUCE-4969
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: 1.2.0, 2.0.3-alpha
Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch,
HADOOP-9176.trunk.patch
RawLocalFileSystem.delete fails on Windows even when the files are not
expected to be in use. It does not reproduce with Sun JDK 6.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4964) JobLocalizer#localizeJobFiles can potentially write job.xml to the wrong user's directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567909#comment-13567909
]
Alejandro Abdelnur commented on MAPREDUCE-4964:
---
Arun, since we applied this patch to the cluster we were seing the issue (2
days ago), this error never occurred again and we don't see any side effects.
Before applying the patch we were seeing this errors on regular basis, errors
typically don't fail because the tasks get retried and eventually the job
completes (we assume this was the reason this was not caught before). We'd like
to get this in ASAP, what are your concerns with the patch?
JobLocalizer#localizeJobFiles can potentially write job.xml to the wrong
user's directory
-
Key: MAPREDUCE-4964
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4964
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: MR-4964.patch, MR-4964.patch
In the following code, if jobs corresponding to different users (X and Y) are
localized simultaneously, it is possible that jobconf can be written to the
wrong user's directory. (X's job.xml can be written to Y's directory)
{code}
public void localizeJobFiles(JobID jobid, JobConf jConf,
Path localJobTokenFile, TaskUmbilicalProtocol taskTracker)
throws IOException, InterruptedException {
localizeJobFiles(jobid, jConf,
lDirAlloc.getLocalPathForWrite(JOBCONF, ttConf), localJobTokenFile,
taskTracker);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4970) Child tasks create security audit log files
Sandy Ryza created MAPREDUCE-4970: - Summary: Child tasks create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, empty log files will be created for tasks with names like SecurityAuth-joeuser.log. I haven't verified whether this occurs in MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4970) Child tasks (try to) create security audit log files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned MAPREDUCE-4970: - Assignee: Sandy Ryza Child tasks (try to) create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, empty log files will be created for tasks with names like SecurityAuth-joeuser.log. I haven't verified whether this occurs in MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4970: -- Summary: Child tasks (try to) create security audit log files (was: Child tasks create security audit log files) Child tasks (try to) create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, empty log files will be created for tasks with names like SecurityAuth-joeuser.log. I haven't verified whether this occurs in MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4970: -- Description: After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, log4j will try to create log files for tasks with names like SecurityAuth-joeuser.log. (was: After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, empty log files will be created for tasks with names like SecurityAuth-joeuser.log. I haven't verified whether this occurs in MR2 yet.) Child tasks (try to) create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, log4j will try to create log files for tasks with names like SecurityAuth-joeuser.log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4710) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568315#comment-13568315 ] Cindy Li commented on MAPREDUCE-4710: - Added patch for trunk. Add peak memory usage counter for each task --- Key: MAPREDUCE-4710 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4710 Project: Hadoop Map/Reduce Issue Type: New Feature Components: task Affects Versions: 1.0.2 Reporter: Cindy Li Assignee: Cindy Li Priority: Minor Labels: patch Attachments: mapreduce-4710.patch, MAPREDUCE-4710-trunk.patch, mapreduce-4710-v1.0.2.patch Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which are snapshots of memory usage of that task. They are not sufficient for users to understand peak memory usage by that task, e.g. in order to diagnose task failures, tune job parameters or change application design. This new feature will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and VIRTUAL_MEMORY_BYTES_MAX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4710) Add peak memory usage counter for each task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cindy Li updated MAPREDUCE-4710: Attachment: MAPREDUCE-4710-trunk.patch Add peak memory usage counter for each task --- Key: MAPREDUCE-4710 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4710 Project: Hadoop Map/Reduce Issue Type: New Feature Components: task Affects Versions: 1.0.2 Reporter: Cindy Li Assignee: Cindy Li Priority: Minor Labels: patch Attachments: mapreduce-4710.patch, MAPREDUCE-4710-trunk.patch, mapreduce-4710-v1.0.2.patch Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which are snapshots of memory usage of that task. They are not sufficient for users to understand peak memory usage by that task, e.g. in order to diagnose task failures, tune job parameters or change application design. This new feature will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and VIRTUAL_MEMORY_BYTES_MAX. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4552) Encryption: Add support for PGP Encryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568325#comment-13568325 ] Benoy Antony commented on MAPREDUCE-4552: - Sure. I'll decompose this into smaller patches. What you mentioned about directory structure is true. If that's going to change, then this feature is going to break. I would need some guidance on this. Once I break it this into smaller patches, we will review that piece separately. Encryption: Add support for PGP Encryption --- Key: MAPREDUCE-4552 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4552 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Reporter: Benoy Antony Assignee: Benoy Antony Attachments: MR_4552_1_1.patch, MR_4552_trunk.patch Provide support for PGP encryption by implementing Encrypter and Decrypter interfaces defined in MAPREDUCE-4450. This can be used by the cluster to protect the job secrets. This also be used map reduce jobs to encrypt/decrypt data. Add PGPCodec as a CompressionCodec so that encrypted data can be processed transparently like compressed data . The aliases to the keys can be specified as part of Job. Based on PGPCodec, a number of utilities are provided to encrypt, decrypt the data in cluster. They include 1.DistributedSplitter – Split an encrypted file into smaller files. 2.DistributedEncrypter – encrypt files in a cluster. 3.DistributedDecrypter – decrypt encrypted files in a cluster. 4.DistributedRecrypter – decrypt an encrypted file and encrypt it with another key. Uitlities are added to encrypt/decrypt files in local file system 1.Genkey - Generate an asymmetric key pair (public and private keys) of a specified strength 2.Encrypt - Encrypt a file 3.Decrypt – Decrypt a file Added as a contrib project - hadoop-crypto. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568338#comment-13568338 ] Sandy Ryza commented on MAPREDUCE-4970: --- I propose creating a task-log4j.properties without the SecurityAuth logger, and passing that instead of the default to the task. This is what MR2 does, with container-log4j.properties. Child tasks (try to) create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, log4j will try to create log files for tasks with names like SecurityAuth-joeuser.log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568346#comment-13568346 ] Karthik Kambatla commented on MAPREDUCE-4970: - +1 on the approach. Child tasks (try to) create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, log4j will try to create log files for tasks with names like SecurityAuth-joeuser.log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4935) Support timeout limitation to MRv1 job end notifications
[ https://issues.apache.org/jira/browse/MAPREDUCE-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Shen updated MAPREDUCE-4935: - Affects Version/s: (was: 0.23.0) 1.0.0 Support timeout limitation to MRv1 job end notifications - Key: MAPREDUCE-4935 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4935 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, mrv1 Affects Versions: 1.0.0 Reporter: Olga Shen Since MAPREDUCE-3028 only added timeout limitation to MRv2 job end notification, please add it to MRv1 job end notification. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3688) Need better Error message if AM is killed/throws exception
[ https://issues.apache.org/jira/browse/MAPREDUCE-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568512#comment-13568512 ] Eli Reisman commented on MAPREDUCE-3688: This has been making me crazy. Please fix this. Every time I install hadoop-2.0.x I spend more time than I'd care to admit fiddling to get this to stop happening: INFO mapreduce.Job: Job job_1359696651870_0001 failed with state FAILED due to: Application application_1359696651870_0001 failed 1 times due to AM Container for appattempt_1359696651870_0001_01 exited with exitCode: 1 due to: .Failing this attempt.. Failing the application. This is not really much to go on. Need better Error message if AM is killed/throws exception -- Key: MAPREDUCE-3688 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3688 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.1 Reporter: David Capwell Assignee: Ravi Prakash Fix For: 0.23.2 We need better error messages in the UI if the AM gets killed or throws an Exception. If the following error gets thrown: java.lang.NumberFormatException: For input string: 9223372036854775807l // last char is an L then the UI should say this exception. Instead I get the following: Application application_1326504761991_0018 failed 1 times due to AM Container for appattempt_1326504761991_0018_01 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
