[jira] [Resolved] (HADOOP-6942) Ability for having user's classes take precedence over the system classes for tasks' classpath
[ https://issues.apache.org/jira/browse/HADOOP-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-6942. - Resolution: Duplicate Fixed via MAPREDUCE-1938. Closing as dupe. Ability for having user's classes take precedence over the system classes for tasks' classpath -- Key: HADOOP-6942 URL: https://issues.apache.org/jira/browse/HADOOP-6942 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 0.22.0 Reporter: Krishna Ramachandran Attachments: HADOOP-6942.y20.patch, hadoop-common-6942.patch Fix bin/hadoop script to facilitate mapred-1938 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8817) Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved HADOOP-8817. - Resolution: Fixed This jira is covered by JIRAs it contains. Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1 Key: HADOOP-8817 URL: https://issues.apache.org/jira/browse/HADOOP-8817 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Labels: features Fix For: 1.2.0 Attachments: HADOOP-8817.patch, HADOOP-8817-v2.patch, HADOOP-8817-v3.patch, HADOOP-8817-v4.patch HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes: 1. Add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470 2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601 3. Update balancer policy:HDFS-3495 4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19. This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9419) CodecPool should avoid OOMs with buggy codecs
[ https://issues.apache.org/jira/browse/HADOOP-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved HADOOP-9419. - Resolution: Won't Fix Never mind. I created a patch, and it is completely useless in fixing this problem. The tasks still OOM because the codec itself is so small and the MergeManager creates new codecs so quickly that on a job with lots of reduces it literally uses up all of the address space with direct byte buffers. Some of the processes get killed by the NM for going over the virtual address space before they OOM. We could try and have the CodecPool detect that the codec is doing the wrong thing and correct it for the codec, but that is too heavy handed in my opinion. CodecPool should avoid OOMs with buggy codecs - Key: HADOOP-9419 URL: https://issues.apache.org/jira/browse/HADOOP-9419 Project: Hadoop Common Issue Type: Improvement Reporter: Robert Joseph Evans I recently found a bug in the gpl compression libraries that was causing map tasks for a particular job to OOM. https://github.com/omalley/hadoop-gpl-compression/issues/3 Now granted it does not make a lot of sense for a job to use the LzopCodec for map output compression over the LzoCodec, but arguably other codecs could be doing similar things and causing the same sort of memory leaks. I propose that we do a sanity check when creating a new decompressor/compressor. If the codec newly created object does not match the value from getType... it should turn off caching for that Codec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7905) Port FileContext symlinks to FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-7905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HADOOP-7905. - Resolution: Duplicate Resolving as dupe of HADOOP-8040 subtasks HADOOP-9414 and HADOOP-9416. Port FileContext symlinks to FileSystem --- Key: HADOOP-7905 URL: https://issues.apache.org/jira/browse/HADOOP-7905 Project: Hadoop Common Issue Type: New Feature Components: fs Reporter: Eli Collins FileSystem isn't going away anytime soon (HADOOP-6446). It would be useful to implement HADOOP-6421 for FileSystem, this would allow interoperability between FileContext and FileSystem (eg currently a symlink created via FileContext is not readable via FileSystem), which will help people migrate to FileContext. The work is mostly moving the client-side link resolution code to a shared place and porting the tests or modifying them to be FC/FS agnostic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
