[jira] [Created] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
Arun C Murthy created MAPREDUCE-5450: Summary: Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 2.1.0-beta We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) - org.apache.hadoop.mapred.IFile$InMemoryReader.init(org.apache.hadoop.mapred.RamManager, org.apache.hadoop.mapred.TaskAttemptID, byte[], int, int) @bci=11, line=480 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createInMemorySegments(java.util.List, long) @bci=133, line=2416 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator() @bci=669, line=2530 (Interpreted frame) -
[jira] [Created] (MAPREDUCE-5408) CLONE - The logging level of the tasks should be configurable by the job
Arun C Murthy created MAPREDUCE-5408: Summary: CLONE - The logging level of the tasks should be configurable by the job Key: MAPREDUCE-5408 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5408 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Owen O'Malley Assignee: Arun C Murthy Fix For: 0.21.0 Attachments: MAPREDUCE-336_branch1.patch It would be nice to be able to configure the logging level of the Task JVM's separately from the server JVM's. Reducing logging substantially increases performance and reduces the consumption of local disk on the task trackers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5260) Job failed because of JvmManager running into inconsistent state
[ https://issues.apache.org/jira/browse/MAPREDUCE-5260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5260. -- Resolution: Fixed Fix Version/s: (was: 1.1.3) 1.2.1 I just committed this. Thanks zhaoyunjiong! Job failed because of JvmManager running into inconsistent state Key: MAPREDUCE-5260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5260 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 1.1.2 Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 1.2.1 Attachments: MAPREDUCE-5260-branch-1.1.patch In our cluster, jobs failed due to randomly task initialization failed because of JvmManager running into inconsistent state and TaskTracker failed to exit: java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:402) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:387) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:192) at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:125) at org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:292) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:251) --- java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.getDetails(JvmManager.java:402) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.reapJvm(JvmManager.java:387) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.access$000(JvmManager.java:192) at org.apache.hadoop.mapred.JvmManager.launchJvm(JvmManager.java:125) at org.apache.hadoop.mapred.TaskRunner.launchJvmAndWait(TaskRunner.java:292) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:251) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5318) Ampersand in JSPUtil.java is not escaped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-5318: -- Ampersand in JSPUtil.java is not escaped Key: MAPREDUCE-5318 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5318 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.1.2 Reporter: Bohou Li Assignee: Bohou Li Priority: Minor Attachments: JSPUtil.class, MAPREDUCE-5318-br1.patch The malformed urls cause hue crash. The malformed urls are caused by the unescaped ampersand . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5318) Ampersand in JSPUtil.java is not escaped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5318. -- Resolution: Fixed Fix Version/s: 1.2.1 I just committed this. Thanks [~bohou]! (In future, please don't resolve a bug yourself unless it gets committed by a committer. Thanks!) Ampersand in JSPUtil.java is not escaped Key: MAPREDUCE-5318 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5318 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.1.2 Reporter: Bohou Li Assignee: Bohou Li Priority: Minor Fix For: 1.2.1 Attachments: JSPUtil.class, MAPREDUCE-5318-br1.patch The malformed urls cause hue crash. The malformed urls are caused by the unescaped ampersand . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5326) Add version to shuffle header
Arun C Murthy created MAPREDUCE-5326: Summary: Add version to shuffle header Key: MAPREDUCE-5326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5326 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker We need to add a version to the shuffle header to allow for forward-compatibility etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5184) Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5184. -- Resolution: Fixed I just committed this. Thanks [~zhijin]! Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2 Key: MAPREDUCE-5184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Reporter: Arun C Murthy Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5184.1.patch, MAPREDUCE-5184.2.patch, MAPREDUCE-5184.3.patch, MAPREDUCE-5184.4.patch, MAPREDUCE-5184.4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5319) Job.xml file does not has 'user.name' property for Hadoop2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5319. -- Resolution: Fixed I just committed this. Thanks [~xgong]! Job.xml file does not has 'user.name' property for Hadoop2 -- Key: MAPREDUCE-5319 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5319 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Fix For: 2.1.0-beta Attachments: MR-5319.2.patch, MR-5319.patch Run a sleep job and look for job.xml file generated by sleep job. It does not contain user.name property. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-5300: -- Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5300) Two function signature changes in filecache.DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5300. -- Resolution: Fixed Hadoop Flags: Incompatible change Re-opened to close with 'Incompatible Change' flag. Two function signature changes in filecache.DistributedCache Key: MAPREDUCE-5300 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5300 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: MAPREDUCE-5300.1.patch Two more incompatibility issues: * long[] getArchiveTimestamps(Configuration) - String[] getArchiveTimestamps(Configuration) * long[] getFileTimestamps(Configuration) - String[] getFileTimestamps(Configuration) Changes will break 0.23 *Move the add-on patch of MAPREDUCE-5263 here.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3859) CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3859. -- Resolution: Fixed Fix Version/s: (was: 2.1.0-beta) I'm resolving this for MR1 since I'll need to open a separate YARN jira for branch-2. Thanks Sergey! CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs Key: MAPREDUCE-3859 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Affects Versions: 1.0.0 Reporter: Sergey Tryuber Assignee: Sergey Tryuber Fix For: 1.2.1 Attachments: MAPREDUCE-3859_MR1_fix_and_test.patch.txt, test-to-fail.patch.txt Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which use 3 map slots will never consume more than 9 slots, regardless how many free slots on a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5290) CLONE - CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
Arun C Murthy created MAPREDUCE-5290: Summary: CLONE - CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs Key: MAPREDUCE-5290 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5290 Project: Hadoop Map/Reduce Issue Type: Bug Components: capacity-sched Affects Versions: 1.0.0 Reporter: Sergey Tryuber Assignee: Sergey Tryuber Fix For: 1.2.1 Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, jobs which use 3 map slots will never consume more than 9 slots, regardless how many free slots on a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4942) mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy reopened MAPREDUCE-4942:
--
Sigh, I missed this.
There are multiple issues with this patch:
# It's incompatible with hadoop-0.23.x
# Has anyone checked Pig and other projects which have already done necessary
to work with hadoop-0.23?
# This badly hoses all users at Yahoo who were brave enough to move to
hadoop-0.23 to test YARN.
The above reasons, plus given the fact that compatibility b/w hadoop-1 for
o.a.h.mapreduce is a non-starter I propose we revert this.
Thoughts?
In future, I'd appreciate a ping if/when major changes to user-facing MR apis
occur - I can provide a lot of context. Thanks.
mapreduce.Job has a bunch of methods that throw InterruptedException so its
incompatible with MR1
-
Key: MAPREDUCE-4942
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4942
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4942.patch
The following methods in MR2's {{org.apache.hadoop.mapreduce.Job}} throw an
{{InterruptedException}} but don't in MR1, which makes them incompatible.
(Their Javadoc comments are also missing that they throw an
{{InterruptedException}} anyway)
I propose that we wrap the {{InterruptedException}} in a
{{RuntimeException}}.
{code}
public float setupProgress() throws IOException, InterruptedException
public float mapProgress() throws IOException, InterruptedException
public float reduceProgress() throws IOException, InterruptedException
public boolean isComplete() throws IOException, InterruptedException
public boolean isSuccessful() throws IOException, InterruptedException
public void killJob() throws IOException, InterruptedException
public void killTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public void failTask(org.apache.hadoop.mapreduce.TaskAttemptID) throws
IOException, InterruptedException
public Counters getCounters() throws IOException, InterruptedException
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5214) Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin
[ https://issues.apache.org/jira/browse/MAPREDUCE-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5214. -- Resolution: Won't Fix Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin - Key: MAPREDUCE-5214 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5214 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 2.0.4-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: mr-5214-v0.patch MRAdmin doesn't apply to MR2. However, to maintain compatibility against 1.x releases, it might be a good idea to add a deprecated version of MRAdmin that wraps around RMAdmin, prints out a deprecated message and calls the relevant RMAdmin methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5223) ant package shouldn't build task-controller by default
Arun C Murthy created MAPREDUCE-5223: Summary: ant package shouldn't build task-controller by default Key: MAPREDUCE-5223 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5223 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy ant package shouldn't build task-controller by default, not sure when we broke this. task-controller should have to be explicitly built via '-Dtask-controller' option. This breaks tar creation on all non-Linux platforms. Sigh. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5064) TestRumenJobTraces failing on 1.3.x and 1.2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-5064.
--
Resolution: Not A Problem
Fixed via an addendum patch to MAPREDUCE-4838.
TestRumenJobTraces failing on 1.3.x and 1.2
---
Key: MAPREDUCE-5064
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5064
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.3.0
Environment: OS/X, java 1.6.0_41, GMT, home network (no DNS)
Reporter: Steve Loughran
Priority: Minor
{{TestRumenJobTraces.testCurrentJHParser()}} is failing locally, both in a
bulk test and standalone
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5206) JT can show the same job multiple times in Retired Jobs section
Arun C Murthy created MAPREDUCE-5206: Summary: JT can show the same job multiple times in Retired Jobs section Key: MAPREDUCE-5206 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5206 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Minor JT can show the same job multiple times in Retired Jobs section since the RetireJobs thread has a bug which adds the same job multiple times to collection of retired jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5198) Race condition in cleanup during task tracker renint with LinuxTaskController
[
https://issues.apache.org/jira/browse/MAPREDUCE-5198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-5198.
--
Resolution: Fixed
Fix Version/s: 1.2.0
I just committed this. Thanks Arpit!
PS: I added a javadoc to the new ttReInit param for TT.TIP.jobHasFinished
during the commit.
Race condition in cleanup during task tracker renint with LinuxTaskController
-
Key: MAPREDUCE-5198
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5198
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 1.2.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
Fix For: 1.2.0
Attachments: MAPREDUCE-5198.patch
This was noticed when job tracker would be restarted while jobs were running
and would ask the task tracker to reinitialize.
Tasktracker would fail with an error like
{code}
013-04-27 20:19:09,627 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred
local directories are:
/grid/0/hdp/mapred/local,/grid/1/hdp/mapred/local,/grid/2/hdp/mapred/local,/grid/3/hdp/mapred/local,/grid/4/hdp/mapred/local,/grid/5/hdp/mapred/local
2013-04-27 20:19:09,628 INFO org.apache.hadoop.ipc.Server: IPC Server handler
3 on 42075 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1717)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:98)
at
org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:744)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:808)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)
2013-04-27 20:19:09,628 INFO org.apache.hadoop.ipc.Server: IPC Server handler
3 on 42075: exiting
2013-04-27 20:19:10,414 ERROR org.apache.hadoop.mapred.TaskTracker: Got fatal
exception while reinitializing TaskTracker:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at
org.apache.hadoop.mapred.LinuxTaskController.deleteAsUser(LinuxTaskController.java:281)
at
org.apache.hadoop.mapred.TaskTracker.deleteUserDirectories(TaskTracker.java:779)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:816)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2704)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3934)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5158) Cleanup required when mapreduce.job.restart.recover is set to false
[ https://issues.apache.org/jira/browse/MAPREDUCE-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5158. -- Resolution: Fixed Fix Version/s: 1.2.0 I just committed this after running affected tests. Thanks Mayank! Cleanup required when mapreduce.job.restart.recover is set to false --- Key: MAPREDUCE-5158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5158 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.2.0 Reporter: yeshavora Assignee: Mayank Bansal Fix For: 1.2.0 Attachments: MAPREDUCE-5158-br1-1.patch, MAPREDUCE-5158-br1.patch When mapred.jobtracker.restart.recover is set as true and mapreduce.job.restart.recover is set to false for a MR job, Job clean up never happens for that job if JT restarts while job is running. .staging and job-info file for that job remains on HDFS forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5184) Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2
Arun C Murthy created MAPREDUCE-5184: Summary: Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2 Key: MAPREDUCE-5184 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5184 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Reporter: Arun C Murthy Assignee: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5131) Provide better handling of job status related apis during JT restart
Arun C Murthy created MAPREDUCE-5131: Summary: Provide better handling of job status related apis during JT restart Key: MAPREDUCE-5131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5131 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy I've seen pig/hive applications bork during JT restart since they get NPEs - this is due to fact that jobs are not really inited, but are submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5131) Provide better handling of job status related apis during JT restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-5131. -- Resolution: Fixed Fix Version/s: 1.2.0 Thanks for the reviews [~szetszwo] and [~kkambatl]. I just committed this. Provide better handling of job status related apis during JT restart Key: MAPREDUCE-5131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5131 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.2.0 Attachments: MAPREDUCE-5131.patch, MAPREDUCE-5131.patch, MAPREDUCE-5131.patch, MAPREDUCE-5131.patch I've seen pig/hive applications bork during JT restart since they get NPEs - this is due to fact that jobs are not really inited, but are submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5108) Review support for Binary Compatibility for MR applications via YARN
Arun C Murthy created MAPREDUCE-5108: Summary: Review support for Binary Compatibility for MR applications via YARN Key: MAPREDUCE-5108 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5108 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Arun C Murthy Assignee: Zhijie Shen Priority: Blocker Fix For: 2.0.5-beta As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly. We've done various pieces of work over time, let's track progress and document things clearly. [~zjshen] has done a bunch of testing and results look very promising so far. The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work. Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible. Also, it's clear that we will *not* be able to support org.apache.hadoop.mapreduce api due to the interface changes we made in hadoop-0.21 (sigh), but given that vast majority of users use the mapred api, it's a reasonable step to easing migration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5109) Job view-acl should apply to job listing too
Arun C Murthy created MAPREDUCE-5109: Summary: Job view-acl should apply to job listing too Key: MAPREDUCE-5109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5109 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Job view-acl should apply to job listing too, currently it only applies to job details pages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4968) Separate MR user apis into a separate module
Arun C Murthy created MAPREDUCE-4968: Summary: Separate MR user apis into a separate module Key: MAPREDUCE-4968 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4968 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy It will be useful to separate MR user apis into a separate module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4943) JobImpl.makeUberDecision needs cleanup
Arun C Murthy created MAPREDUCE-4943: Summary: JobImpl.makeUberDecision needs cleanup Key: MAPREDUCE-4943 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4943 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy JobImpl.makeUberDecision needs cleanup: # Uses hard-coded default values in lots of places # Need to fix it to use block-size of input while checking input-data # Need to stop using JobConf.DISABLED_MEMORY_LIMIT # Could use a real unit test -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4520) Add experimental support for MR AM to schedule CPUs along-with memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4520. -- Resolution: Fixed Fix Version/s: 2.0.3-alpha I just committed this trivial patch post YARN-2. Add experimental support for MR AM to schedule CPUs along-with memory - Key: MAPREDUCE-4520 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4520 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4520.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4918) Better error message in TrackerDistributedCacheManager.ancestorsHaveExecutePermissions
Arun C Murthy created MAPREDUCE-4918: Summary: Better error message in TrackerDistributedCacheManager.ancestorsHaveExecutePermissions Key: MAPREDUCE-4918 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4918 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Xuan Gong Better logging/error message in TrackerDistributedCacheManager.ancestorsHaveExecutePermissions should help debugging (e.g. MAPREDUCE-4916). We should log the offending parent directory with the incorrect permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4916) TestTrackerDistributedCacheManager is flaky due to other badly written tests
Arun C Murthy created MAPREDUCE-4916: Summary: TestTrackerDistributedCacheManager is flaky due to other badly written tests Key: MAPREDUCE-4916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4916 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Xuan Gong Credit to Xuan figuring this: TestTrackerDistributedCacheManager is flaky due to other badly written tests since it checks for existence of a directory upfront which might have bad perms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4869) TestMapReduceChildJVM fails in branch-trunk-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-4869.
--
Resolution: Fixed
Fix Version/s: trunk-win
I just committed this. Thanks Chris!
TestMapReduceChildJVM fails in branch-trunk-win
---
Key: MAPREDUCE-4869
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4869
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: trunk-win
Attachments: MAPREDUCE-4869-branch-trunk-win.1.patch
The YARN-233 patch for getting YARN working on Windows forgot to include a
corresponding change in {{TestMapReduceChildJVM}}, so the test is failing now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4888) NLineInputFormat drops data in 1.1 and beyond
[ https://issues.apache.org/jira/browse/MAPREDUCE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4888. -- Resolution: Fixed Fix Version/s: 1.1.2 I just committed this. Thanks Vinod and Bobby! NLineInputFormat drops data in 1.1 and beyond - Key: MAPREDUCE-4888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4888 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Robert Joseph Evans Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 1.1.2 Attachments: MAPREDUCE-4888-20121219.txt, test.patch When trying to root cause why MAPREDUCE-4782 did not cause us issues on 1.0.2, I found out that HADOOP-7823 introduced essentially the exact same error into org.apache.hadoop.mapred.lib.NLineInputFormat. In 1.X org.apache.hadoop.mapred.lib.NLineInputFormat and org.apache.hadoop.mapreduce.lib.input.NLineInputFormat are separate implementations. The latter had an off by one error in it until MAPREDUCE-4782 fixed it. The former had no error in it until HADOOP-7823 introduced it in 1.1 and MAPREDUCE-375 combined the implementations together but picked the implementation with the off by one error in 0.21. I will attach a patch that exposes the error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4891) Pluggable merge at reduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4891. -- Resolution: Duplicate Duplicate of MAPREDUCE-4808 Pluggable merge at reduce side -- Key: MAPREDUCE-4891 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4891 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Jerry Chen Original Estimate: 504h Remaining Estimate: 504h The current implementation of sort in MapReduce is cooperated by Map side sort and Reduce side merge. MAPREDUCE-2454 provided pluggable sort at the Map side currently and pluggable shuffle at Reduce side, while no pluggable merger provided. Considering a general need of hash grouping and join, we may need to replace both the Map Sort and Reduce merge with a more light weight hash grouping alorithm. A general pluggable merge would help support this need. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2454. -- Resolution: Fixed Fix Version/s: 3.0.0 Thanks Tucu. Closing this. Allow external sorter plugin for MR --- Key: MAPREDUCE-2454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Priority: Minor Labels: features, performance, plugin, sort Fix For: 3.0.0 Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, mapreduce-2454-modified-code.patch, mapreduce-2454-modified-test.patch, mapreduce-2454-new-test.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454-protection-change.patch, mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java Define interfaces and some abstract classes in the Hadoop framework to facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4812) Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4812. -- Resolution: Duplicate Fix Version/s: (was: 2.0.3-alpha) For now I'll close this as a dup of MAPREDUCE-4808, we are having same discussions in both places Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle --- Key: MAPREDUCE-4812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Attachments: COMBO-mapreduce-4809-4812.patch, COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch This is part of MAPREDUCE-2454. This further breaks down MAPREDUCE-4808 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1
Arun C Murthy created MAPREDUCE-4859: Summary: TestRecoveryManager fails on branch-1 Key: MAPREDUCE-4859 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.1.2 Looks like the tests are extremely flaky and just hang. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4859. -- Resolution: Fixed I just committed this. Thanks for the quick check Matt! TestRecoveryManager fails on branch-1 - Key: MAPREDUCE-4859 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.1.2 Attachments: MAPREDUCE-4859.patch Looks like the tests are extremely flaky and just hang. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4854) TestRumenJobTraces is broken in branch-1
Arun C Murthy created MAPREDUCE-4854: Summary: TestRumenJobTraces is broken in branch-1 Key: MAPREDUCE-4854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4854 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.1.2 TestRumenJobTraces is broken in branch-1, need to fix the 'gold' events it's checking against which is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4854) TestRumenJobTraces is broken in branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4854. -- Resolution: Cannot Reproduce Sorry, looks like an error - looking at wrong branch. TestRumenJobTraces is broken in branch-1 Key: MAPREDUCE-4854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4854 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.0 Reporter: Arun C Murthy Assignee: Arun C Murthy TestRumenJobTraces is broken in branch-1, need to fix the 'gold' events it's checking against which is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4858) TestWebUIAuthorization fails on branch-1
Arun C Murthy created MAPREDUCE-4858: Summary: TestWebUIAuthorization fails on branch-1 Key: MAPREDUCE-4858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4858 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy TestWebUIAuthorization fails on branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4858) TestWebUIAuthorization fails on branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4858. -- Resolution: Fixed Fix Version/s: 1.1.2 Thanks Mahadev Vinod. I committed this for Matt to pick up for 1.1.2. TestWebUIAuthorization fails on branch-1 Key: MAPREDUCE-4858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4858 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 1.1.2 Attachments: MAPREDUCE-4858.patch, MAPREDUCE-4858.patch TestWebUIAuthorization fails on branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4837) Add MR-AM web-services to branch-1
Arun C Murthy created MAPREDUCE-4837: Summary: Add MR-AM web-services to branch-1 Key: MAPREDUCE-4837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4837 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Add MR-AM web-services to branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4838) Add extra info to JH files
Arun C Murthy created MAPREDUCE-4838: Summary: Add extra info to JH files Key: MAPREDUCE-4838 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4838 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy It will be useful to add more task-info to JH for analytics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4682) TestKillSubProcess TestTaskTrackerMemoryManager fail to compile on trunk due to MAPREDUCE-4253
[ https://issues.apache.org/jira/browse/MAPREDUCE-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4682. -- Resolution: Not A Problem Not a problem, I've reverted the original commit. TestKillSubProcess TestTaskTrackerMemoryManager fail to compile on trunk due to MAPREDUCE-4253 Key: MAPREDUCE-4682 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4682 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fail with: /Users/acmurthy/dev/apache/hadoop/hadoop-trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestKillSubProcesses.java:411: cannot find symbol [javac] symbol : variable TestProcfsBasedProcessTree [javac] location: class org.apache.hadoop.mapred.TestKillSubProcesses [javac] childPid = TestProcfsBasedProcessTree.getPidFromPidFile(scriptDirName [javac]^ [javac] /Users/acmurthy/dev/apache/hadoop/hadoop-trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskTrackerMemoryManager.java:449: cannot find symbol [javac] symbol : variable TestProcfsBasedProcessTree [javac] location: class org.apache.hadoop.mapred.TestTaskTrackerMemoryManager [javac] TestProcfsBasedProcessTree.setupProcfsRootDir(procfsRootDir); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4687) Add compilation of 'classic' MR1 (ant based) to jenkins builds
Arun C Murthy created MAPREDUCE-4687: Summary: Add compilation of 'classic' MR1 (ant based) to jenkins builds Key: MAPREDUCE-4687 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4687 Project: Hadoop Map/Reduce Issue Type: Bug Environment: Adding compilation of 'classic' MR1 (ant based) to jenkins builds will help prevent situations like MAPREDUCE-4681 MAPREDUCE-4682. Reporter: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4564) Shell timeout mechanism does not work for processes spawned using winutils
[ https://issues.apache.org/jira/browse/MAPREDUCE-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4564. -- Resolution: Fixed Fix Version/s: 1-win I just committed this. Thanks Bikas (and Chuan for the review). Shell timeout mechanism does not work for processes spawned using winutils -- Key: MAPREDUCE-4564 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4564 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1-win Attachments: MAPREDUCE-4564.branch-1-win.1.patch, MAPREDUCE-4564.branch-1-win.2.patch Upon timeout, Shell calls Java process.destroy() to terminate the spawned process. This would destroy the winutils process but not the real process spawned by winutils. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4613) Scheduling of reduce tasks results in starvation
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4613. -- Resolution: Duplicate Dup of MAPREDUCE-4299. Thanks Vasco. Scheduling of reduce tasks results in starvation Key: MAPREDUCE-4613 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler Affects Versions: 0.23.1, 2.0.1-alpha Environment: 16 (duo core) machine cluster == 32 containers namenode and resourcemanager running on separate 17th machine Reporter: Vasco Attachments: scheduling.png If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo. I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be pre-empted. However, pre-emption does not occur. A work-around is to set the number of reducers available containers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4603) Allow JobClient to retry job-submission when JT is in safemode
Arun C Murthy created MAPREDUCE-4603: Summary: Allow JobClient to retry job-submission when JT is in safemode Key: MAPREDUCE-4603 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4603 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Similar to HDFS-3504, it would be useful to allow JobClient to retry job-submission when JT is in safemode (via MAPREDUCE-4328). This way applications like Pig/Hive don't bork midway when the NN/JT are not operational. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4605) Text File Busy errors launching MR tasks
Arun C Murthy created MAPREDUCE-4605:
Summary: Text File Busy errors launching MR tasks
Key: MAPREDUCE-4605
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4605
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Andy Isaacson
Fix For: 1.2.0, 0.23.3, 2.2.0-alpha
Some very small percentage of tasks fail with a Text file busy error.
The following was the original diagnosis:
{quote}
Our use of PrintWriter in TaskController.writeCommand is unsafe, since that
class swallows all IO exceptions. We're not currently checking for errors,
which I'm seeing result in occasional task failures with the message Text file
busy - assumedly because the close() call is failing silently for some reason.
{quote}
.. but turned out to be another issue as well (see below)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4374) Fix child task environment variable config and add support for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4374. -- Resolution: Fixed Fix Version/s: 1-win I just committed this. Thanks Chuan for the patch, and Bikas for the review. Fix child task environment variable config and add support for Windows -- Key: MAPREDUCE-4374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4374 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor Fix For: 1-win Attachments: MAPREDUCE-4374-branch-1-win-2.patch, MAPREDUCE-4374-branch-1-win.patch In HADOOP-2838, a new feature was introduced to set environment variables via the Hadoop config 'mapred.child.env' for child tasks. There are some further fixes and improvements around this feature, e.g. HADOOP-5981 were a bug fix; MAPREDUCE-478 broke the config into 'mapred.map.child.env' and 'mapred.reduce.child.env'. However the current implementation is still not complete. It does not match its documentation or original intend as I believe. Also, by using ‘:’ (colon) and ‘;’ (semicolon) in the configuration syntax, we will have problems using them on Windows because ‘:’ appears very often in Windows path as in “C:\”, and environment variables are used very often to hold path names. The Jira is created to fix the problem and provide support on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4575) Add an option to drain the JobTracker jobs for upgrades
Arun C Murthy created MAPREDUCE-4575: Summary: Add an option to drain the JobTracker jobs for upgrades Key: MAPREDUCE-4575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4575 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Following on from MAPREDUCE-4328 it will be useful to allow an option to drain the JobTracker so that it will finish up existing jobs and not accept new ones. This is particularly useful during upgrades. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3410) Capacity scheduler reconfiguration of queues does not work for add sub-queues to an existing queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3410. -- Resolution: Later For now we should punt on this - this is akin to 'delete queue' as Jason explained. Capacity scheduler reconfiguration of queues does not work for add sub-queues to an existing queue -- Key: MAPREDUCE-3410 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3410 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth If we have an existing queue configuration such as root |--- A |--- B and we attempt to reconfigure it so that we now have root |--- A |--- A1 |--- A2 |--- B we get an IOException as follows: java.io.IOException: Failed to re-init queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:197) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue.testInitializeQueue(TestLeafQueue.java:206) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:45) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:172) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:78) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:70) Caused by: java.io.IOException: Trying to reinitialize root.a from root.a at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.reinitialize(LeafQueue.java:524) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.reinitialize(ParentQueue.java:360) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:240) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:194) ... 32 more This is apparently because the CapacityScheduler still wants to think of A as a LeafQueue instead of realizing it to be updated as a ParentQueue. Maybe, this use case is not supposed to be supported, in which case, probably the documentation should be updated to state this scenario as such more clearly
[jira] [Created] (MAPREDUCE-4476) Add more unit tests for CPU scheduling in CS
Arun C Murthy created MAPREDUCE-4476: Summary: Add more unit tests for CPU scheduling in CS Key: MAPREDUCE-4476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4476 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Assignee: Arun C Murthy Companion to MAPREDUCE-4327. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4340) Node Manager leaks socket connections connected to Data Node
[ https://issues.apache.org/jira/browse/MAPREDUCE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4340. -- Resolution: Duplicate Duplicate of MAPREDUCE-4323 Node Manager leaks socket connections connected to Data Node Key: MAPREDUCE-4340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4340 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Priority: Critical I am running simple wordcount example with default configurations, for every job run it increases one datanode socket connection and it will be there in CLOSE_WAIT state forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration
[ https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-4422: -- Uh, Harsh/Ahmed, I'm confused... the newly added variable DEFAULT_YARN_APPLICATION_CLASSPATH isn't being used anywhere? The point of adding this was to use it, at least, in MRApps.setMRFrameworkClasspath? Also, not sure whether this was introduced via MAPREDUCE-3505, but we should use Configuration.getStrings rather than do a 'manual' String.split to parse this out. YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration Key: MAPREDUCE-4422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422 Project: Hadoop Map/Reduce Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Assignee: Ahmed Radwan Fix For: 2.0.1-alpha Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable. However, we didn't add a default value to YarnConfiguration, as-is the norm. Ran into it while investigating MAPREDUCE-4421. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4439) MAPREDUCE-3451 introduced a bunch of findbugs warnings
Arun C Murthy created MAPREDUCE-4439: Summary: MAPREDUCE-3451 introduced a bunch of findbugs warnings Key: MAPREDUCE-4439 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4439 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4440) Change SchedulerApp to be a minimal interface
Arun C Murthy created MAPREDUCE-4440: Summary: Change SchedulerApp to be a minimal interface Key: MAPREDUCE-4440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4440 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.1-alpha Reporter: Arun C Murthy Fix For: 2.0.1-alpha Schedulers should manage their own implementations of SchedulerApp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3837) Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-3837: -- Looks like this needs a minor update to get it to work on Mac OSX... Job tracker is not able to recover job in case of crash and after that no user can submit job. -- Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 0.22.0, 1.1.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.24.0, 1.2.0, 0.22.1, 0.23.2 Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837-3.patch, PATCH-HADOOP-1-MAPREDUCE-3837-4.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3837) Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3837. -- Resolution: Fixed Thanks for the reviews Tom Mayank. I've just committed the small patch. Job tracker is not able to recover job in case of crash and after that no user can submit job. -- Key: MAPREDUCE-3837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 0.22.0, 1.1.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 1.2.0, 0.22.1 Attachments: MAPREDUCE-3837_addendum.patch, PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837-3.patch, PATCH-HADOOP-1-MAPREDUCE-3837-4.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch If job tracker is crashed while running , and there were some jobs are running , so if job tracker's property mapreduce.jobtracker.restart.recover is true then it should recover the job. However the current behavior is as follows jobtracker try to restore the jobs but it can not . And after that jobtracker closes its handle to hdfs and nobody else can submit job. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4421) Remove dependency on deployed MR jars
Arun C Murthy created MAPREDUCE-4421: Summary: Remove dependency on deployed MR jars Key: MAPREDUCE-4421 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Currently MR AM depends on MR jars being deployed on all nodes via implicit dependency on YARN_APPLICATION_CLASSPATH. We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, probably, just rely on adding a shaded MR jar along with job.jar to the dist-cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration
Arun C Murthy created MAPREDUCE-4422: Summary: YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration Key: MAPREDUCE-4422 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable. However, we didn't add a default value to YarnConfiguration, as-is the norm. Ran into it while investigating MAPREDUCE-4421. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4236) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4236. -- Resolution: Not A Problem Mostly an env. issue. Closing for now. Failing tests in branch-2 - Key: MAPREDUCE-4236 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4236 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Priority: Critical Running org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.872 sec FAILURE! -- Running org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.862 sec FAILURE! -- Running org.apache.hadoop.conf.TestNoDefaultsJobConf Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.865 sec FAILURE! -- Running org.apache.hadoop.mapreduce.security.TestJHSSecurity Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.372 sec FAILURE! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3998) taskjvm.sh: Permission denied
[ https://issues.apache.org/jira/browse/MAPREDUCE-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3998. -- Resolution: Invalid This looks like a CDH issue. taskjvm.sh: Permission denied - Key: MAPREDUCE-3998 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3998 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2 Environment: uname -a : 2.6.18-194.17.4.0.1.el5PAE #1 SMP Tue Oct 26 20:15:18 EDT 2010 i686 i686 i386 GNU/Linux Reporter: toughman Priority: Critical run a simple code under cdh3u3, the slave node's map task and reduce task failed, this is the error info from the tasktracker's log: 2012-03-09 17:25:56,562 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 126 2012-03-09 17:25:56,563 WARN org.apache.hadoop.mapred.DefaultTaskController: Task wrapper stderr: bash: /home/mapred/local/ttprivate/taskTracker/hdfs/jobcache/job_201203091543_0003/attempt_201203091543_0003_m_02_0/taskjvm.sh: Permission denied __I think the privilige of the /home/mapred/local/ is properly set -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4409) Add support for FifoScheduler to schedule CPU along with memory.
Arun C Murthy created MAPREDUCE-4409: Summary: Add support for FifoScheduler to schedule CPU along with memory. Key: MAPREDUCE-4409 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4409 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4410) Add experimental support for MR AM to schedule CPUs along-with memory
Arun C Murthy created MAPREDUCE-4410: Summary: Add experimental support for MR AM to schedule CPUs along-with memory Key: MAPREDUCE-4410 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4410 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4411) Add support for DistributedShell to ask for CPUs along with memory
Arun C Murthy created MAPREDUCE-4411: Summary: Add support for DistributedShell to ask for CPUs along with memory Key: MAPREDUCE-4411 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4411 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4322) Fix command-line length abort issues on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-4322.
--
Resolution: Fixed
Fix Version/s: 1-win
Target Version/s: (was: 1.1.0)
+1, lgtm.
I just committed this. Thanks Ivan for the patch and Bikas for the detailed
reviews.
Fix command-line length abort issues on Windows
---
Key: MAPREDUCE-4322
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4322
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Environment: Windows, downstream applications with long aggregate
classpaths
Reporter: John Gordon
Assignee: Ivan Mitic
Fix For: 1-win
Attachments: MAPREDUCE-4322-branch-1-win(2).patch,
MAPREDUCE-4322-branch-1-win(3).patch, MAPREDUCE-4322-branch-1-win(4).patch,
MAPREDUCE-4322-branch-1-win(5).patch, MAPREDUCE-4322-branch-1-win.patch
Original Estimate: 12h
Remaining Estimate: 12h
When a task is started on the tasktracker, it creates a small batch file to
invoke java and runs that batch. Within the batch file, the invocation of
Java currently has -classpath ${CLASSPATH} inline to the command. That line
often exceeds 8000 characters. This is ok for most linux distributions
because the line limit env variable is often set much higher than this.
However, for Windows this cause cmd to abort execution. This surfaces in
Hadoop as an unknown failure mode for the task.
I think the easiest and most natural way to fix this is to push the
-classpath option into a config file to take the longest variable part of the
line and put it somewhere that scales better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin
[ https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4369. -- Resolution: Fixed Fix Version/s: 1-win I just committed this. Thanks Bikas! Fix streaming job failures with WindowsResourceCalculatorPlugin --- Key: MAPREDUCE-4369 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1-win Attachments: MAPREDUCE-4369.branch-1-win.1.patch, MAPREDUCE-4369.branch-1-win.2.patch Some streaming jobs use local mode job runs that do not start tasks trackers. In these cases, the jvm context is not setup and hence local mode execution causes the code to crash. Fix is to not not use ResourceCalculatorPlugin in such cases or make the local job run creating dummy jvm contexts. Choosing the first option because thats the current implicit behavior in Linux. The ProcfsBasedProcessTree (used inside the LinuxResourceCalculatorPlugin) does no real work when the process pid is not setup correctly. This is what happens when local job mode runs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4368) TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4368. -- Resolution: Fixed Fix Version/s: 1-win Looks good, +1. I just committed this. Thanks for the patch John and Bikas/Ivan for reviews. TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces - Key: MAPREDUCE-4368 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 1-win Environment: on Windows: set PATH=%PATH%;C:\this memorable place. Reporter: John Gordon Assignee: John Gordon Labels: newbie, patch Fix For: 1-win Attachments: TaskRunner.patch, TaskRunner2.patch, TaskRunner3.patch Original Estimate: 24h Remaining Estimate: 24h TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with -- so you get something like java -Dopt=value. This is problematic for paths with embedded spaces, where we end up creating -Dopt=first part last part. To java, the jar being run is last part. So with the environment above, you will see ClassNoDefFoundError: memorable and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-4355: -- I'm sorry, but we *cannot* make an incompatible change to JobClient which is a public API, at least in hadoop-1.x -1 on this change. This will break a number of existing apis. It seems we cud just add start-time to RunningJob if necessary. Alejandro - do you mind reverting this change since it breaks compatibility? Thanks. Add JobStatus getJobStatus(JobID) to JobClient. --- Key: MAPREDUCE-4355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv1, mrv2 Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 1.1.0, 2.0.1-alpha Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch To read the start-time of a particular job, one should not need to getAllJobs() and iterate through them. getJob(JobID) returns RunningJob, which doesn't hold the job's start time. Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4203. -- Resolution: Fixed Fix Version/s: 1-win Release Note: Added an implementation of the process tree for Windows. I just committed this. Thanks Bikas! Create equivalent of ProcfsBasedProcessTree for Windows --- Key: MAPREDUCE-4203 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 1-win Attachments: MAPREDUCE-4203.branch-1-win.1.patch, MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4345) ZK-based High Availability (HA) for ResourceManager (RM)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-4345.
--
Resolution: Duplicate
Duplicate of MAPREDUCE-4326.
ZK-based High Availability (HA) for ResourceManager (RM)
Key: MAPREDUCE-4345
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4345
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Harsh J
Assignee: Bikas Saha
One of the goals presented on MAPREDUCE-279 was to have high availability.
One way that was discussed, per Mahadev/others on
https://issues.apache.org/jira/browse/MAPREDUCE-2648 and other places, was ZK:
{quote}
Am not sure, if you already know about the MR-279 branch (the next version of
MR framework). We've been trying to integrate ZK into the framework from the
beginning. As for now, we are just doing restart with ZK but soon we should
have a HA soln with ZK.
{quote}
There is now MAPREDUCE-4343 that tracks recoverability via ZK. This JIRA is
meant to track HA via ZK.
Currently there isn't a HA solution for RM, via ZK or otherwise.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4343) ZK recovery support for ResourceManager
[
https://issues.apache.org/jira/browse/MAPREDUCE-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-4343.
--
Resolution: Duplicate
Duplicate of MAPREDUCE-4326.
ZK recovery support for ResourceManager
---
Key: MAPREDUCE-4343
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4343
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Harsh J
Attachments: MR-4343.1.patch
MAPREDUCE-279 included bits and pieces of possible ZK integration for YARN's
RM, but looks like it failed to complete it (for scalability reasons? etc?)
and there seems to be no JIRA tracking this feature that has been already
claimed publicly as a good part about YARN.
If it did complete it, we should document how to use it. Setting the
following only yields:
{code}
property
nameyarn.resourcemanager.store.class/name
valueorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore/value
/property
property
nameyarn.resourcemanager.zookeeper-store.address/name
valuetest.vm:2181/yarn-recovery-store/value
/property
{code}
{code}
Error starting ResourceManager
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore.init()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:128)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.StoreFactory.getStore(StoreFactory.java:32)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:621)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKStore.init()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:122)
... 2 more
{code}
This JIRA is hence filed to track the addition/completion of recovery via ZK.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
Arun C Murthy created MAPREDUCE-4334: Summary: Add support for CPU isolation/monitoring of containers Key: MAPREDUCE-4334 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Once we get in MAPREDUCE-4327, it will be important to actually enforce limits on CPU consumption of containers. Several options spring to mind: # taskset (RHEL5+) # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4321) DefaultTaskController fails to launch tasks on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-4321.
--
Resolution: Fixed
Target Version/s: 1-win
I just committed this. Thanks Ivan for the patch and Bikas for the review.
DefaultTaskController fails to launch tasks on Windows
--
Key: MAPREDUCE-4321
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4321
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: MAPREDUCE-4321-branch-1-win.patch
DefaultTaskController#launchTask tries to run the child JVM task with the
following command line:
{code}cmd.exe /c /c:/some/path.../taskjvm.cmd{code}
And this fails because the given path is prefixed with a forward slash. This
also causes a number of tests to fail:
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.fs.TestCopyFiles
org.apache.hadoop.mapred.TestBadRecords
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs
org.apache.hadoop.mapred.TestControlledMapReduceJob
org.apache.hadoop.mapred.TestCustomOutputCommitter
org.apache.hadoop.mapred.TestEmptyJob
org.apache.hadoop.mapred.TestFileOutputFormat
org.apache.hadoop.mapred.TestIsolationRunner
org.apache.hadoop.mapred.TestJavaSerialization
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapred.TestJobHistoryServer
org.apache.hadoop.mapred.TestJobInProgressListener
org.apache.hadoop.mapred.TestJobKillAndFail
org.apache.hadoop.mapred.TestJobName
...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4328) Add the option to quiesce the JobTracker
Arun C Murthy created MAPREDUCE-4328: Summary: Add the option to quiesce the JobTracker Key: MAPREDUCE-4328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4328 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 1.0.3 Reporter: Arun C Murthy Assignee: Arun C Murthy In several failure scenarios it would be very handy to have an option to quiesce the JobTracker. Recently, we saw a case where the NameNode had to be rebooted at a customer due to a random hardware failure - in such a case it would have been nice to not lose jobs by quiescing the JobTracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4326) Resurrect RM Restart
Arun C Murthy created MAPREDUCE-4326: Summary: Resurrect RM Restart Key: MAPREDUCE-4326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4326 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Assignee: Bikas Saha We should resurrect 'RM Restart' which we disabled sometime during the RM refactor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
Arun C Murthy created MAPREDUCE-4327: Summary: Enhance CS to schedule accounting for both memory and cpu cores Key: MAPREDUCE-4327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2, resourcemanager, scheduler Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Assignee: Arun C Murthy With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4316) Improve default config values for YARN
Arun C Murthy created MAPREDUCE-4316: Summary: Improve default config values for YARN Key: MAPREDUCE-4316 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240. This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4284) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-job basis
[ https://issues.apache.org/jira/browse/MAPREDUCE-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4284. -- Resolution: Invalid Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-job basis Key: MAPREDUCE-4284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4284 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Ahmed Radwan Assignee: Ahmed Radwan The yarn.nodemanager.delete.debug-delay-sec property is helpful in debugging jobs (inspecting container logs/local dirs after the job finishes). Currently it is a nodemanager property and changing it requires restarting the nodemanager. In a production cluster this can be a real problem. It is better to have this property set on a per-job basis and not requiring the restart of nodemanagers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-2911: -- Hamster: Hadoop And Mpi on the same cluSTER --- Key: MAPREDUCE-2911 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Environment: All Unix-Environments Reporter: Milind Bhandarkar Assignee: Ralph H Castain Original Estimate: 336h Remaining Estimate: 336h MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2911. -- Resolution: Later Fix Version/s: (was: 0.24.0) bq. I was curious to see the code. Which branch has it been committed to? To clarify, there was no code committed to Hadoop itself. Ralph - Can you please share information on where it was committed to OpenMPI and how folks can try it out? Thanks. Hamster: Hadoop And Mpi on the same cluSTER --- Key: MAPREDUCE-2911 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Environment: All Unix-Environments Reporter: Milind Bhandarkar Assignee: Ralph H Castain Original Estimate: 336h Remaining Estimate: 336h MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3090) Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere
Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere --- Key: MAPREDUCE-3090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3090 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 Change MR AM to use ApplicationAttemptId rather than applicationId, startCount everywhere, particularly after MAPREDUCE-3055 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3052) Maintain consistency in naming appIDs, jobIDs and attemptIDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3052. -- Resolution: Duplicate Fix Version/s: (was: 0.24.0) (was: 0.23.0) Duplicate of MAPREDUCE-2793 Maintain consistency in naming appIDs, jobIDs and attemptIDs Key: MAPREDUCE-3052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3052 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Jeffrey Naisbitt Assignee: Jeffrey Naisbitt Currently, the appIDs, jobIDs and attempt/container ids are not consistently named in the logs, console and UI. Some recent jiras have fixed the inconsistencies with the appID. For jobID On the RM UI: job_1308259676864_5_5 JHS UI: job_1308259676864_5_5 Console/logs: job_1308259676864_0005 mapred-local dirs are named as: No jobID I am planning on changing the jobID to match job_1308259676864_0005 in the RM UI and the JHS UI. For attemptID On the RM UI: attempt_1308259676864_5_5_m_24_0 JHS attempt_1308259676864_5_5_m_24_0 Console/logs: attempt_1308259676864_0005_m_24_0 mapred-local dirs are named as: container_1308259676864_0005_24 I'm not sure the best way to handle the attempt and container IDs, but I'm making them more consistent. (If any of you have preferences, let me know) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3089) Re-enable TestRMContainerAllocator
Re-enable TestRMContainerAllocator -- Key: MAPREDUCE-3089 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3089 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Priority: Blocker Fix For: 0.23.0 Re-enable TestRMContainerAllocator to ensure we have good coverage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1207) Allow admins to set java options for map/reduce tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-1207.
--
Resolution: Fixed
Assignee: Arun C Murthy (was: Krishna Ramachandran)
Fixed in trunk/branch-0.23 via MAPREDUCE-2880.
Allow admins to set java options for map/reduce tasks
-
Key: MAPREDUCE-1207
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1207
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, mrv2
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Priority: Blocker
Attachments: mapred-1207.patch
It will be useful for allow cluster-admins to set some java options for child
map/reduce tasks.
E.g. We've had to ask users to set -Djava.net.preferIPv4Stack=true in their
jobs, it would be nice to do it for all users in such scenarios even when
people override mapred.child.{map|reduce}.java.opts but forget to add this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2748) [MR-279] NM should pass a whitelisted environmental variables to the container
[ https://issues.apache.org/jira/browse/MAPREDUCE-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2748. -- Resolution: Fixed Assignee: Arun C Murthy Fixed via MAPREDUCE-2880. [MR-279] NM should pass a whitelisted environmental variables to the container --- Key: MAPREDUCE-2748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Vinod Kumar Vavilapalli Assignee: Arun C Murthy Fix For: 0.23.0 This is similar to [MAPREDUCE-103] . We should pass a whitelisted set of environment variables from NM env to the container. By default, we should pass HADOOP_* variables. This can be a simple configuration key that NodeManager reads. Today, we already either pass the following correctly or assume that it works but doesn't - YARN_HOME: ContainerLaunch#writeLaunchEnv - HADOOP_CLIENT_OPTS: MapReduceChildJVM#setVMEnv - JAVA_HOME: TaskAttemptImpl#createContainerLaunchContext - Works by shell-expansion. - LD_LIBRARY_PATH: Assumed to work via shell-expansion but doesn't. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2717) Client should be able to know why an AM crashed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2717. -- Resolution: Duplicate Assignee: (was: Siddharth Seth) Most are fixed, now the diagnostics part is dup of MAPREDUCE-3065 Client should be able to know why an AM crashed. Key: MAPREDUCE-2717 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2717 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Amol Kekre Priority: Blocker Fix For: 0.23.0 Today if an AM crashes, we have to dig through logs - very cumbersome. It is good to have client print some reason for AM crash. Various possible reasons for AM crash: (1) AM container failed during localization itself. (2) AM container launched but failed before properly starting, for e.g. due to classpath issues (3) AM failed after starting properly. (4) an AM is expired and killed by the RM Potential fixes: - For (1) and (2) the client should obtain the container-status, container diagnostics and exit code. - For (3), the AM should set some kind of reason for failure during its heartbeat to RM and the client should obtain the same from RM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3076) TestSleepJob fails
TestSleepJob fails --- Key: MAPREDUCE-3076 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3076 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.20.205.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Fix For: 0.20.205.0 Attachments: MAPREDUCE-3076.patch TestSleepJob fails, it was intended to be used in other tests for MAPREDUCE-2981. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2909) Docs for remaining records in yarn-api
[ https://issues.apache.org/jira/browse/MAPREDUCE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2909. -- Resolution: Fixed Assignee: Arun C Murthy Fixed via MAPREDUCE-2896 MAPREDUCE-2899 Docs for remaining records in yarn-api -- Key: MAPREDUCE-2909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2909 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.0 MAPREDUCE-2891 , MAPREDUCE-2897 MAPREDUCE-2898 added javadocs for core protocols (i.e. AMRMProtocol, ClientRMProtocol ContainerManager). Most 'records' also have javadocs - this jira is to track the remaining ones. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2718) Job fails if AppMaster is killed
[
https://issues.apache.org/jira/browse/MAPREDUCE-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-2718.
--
Resolution: Not A Problem
Job fails if AppMaster is killed
Key: MAPREDUCE-2718
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2718
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Amol Kekre
Fix For: 0.23.0
Started a cluster. Sumitted a sleep job with around 1 maps and 1000
reduces.
when 5000 maps got completed, It killed AppMaster.
RM web UI Application as failed.
And jobclient after retry for 50 times -:
{
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:161)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:254)
at
org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:520)
at
org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:540)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1130)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1084)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:259)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
Caused by: com.google.protobuf.ServiceException: java.net.ConnectException:
Call to /98.137.103.174:42557 failed on
connection exception: java.net.ConnectException: Connection refused
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:96)
at $Proxy11.getTaskAttemptCompletionEvents(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBClientImpl.java:154)
... 21 more
Caused by: java.net.ConnectException: Call to /... failed on connection
exception:
java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1087)
at org.apache.hadoop.ipc.Client.call(Client.java:1063)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:250)
at org.apache.hadoop.yarn.ipc.$Proxy10.call(Unknown Source)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:94)
... 23 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:375)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:448)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:536)
at
org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:211)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196)
at org.apache.hadoop.ipc.Client.call(Client.java:1040)
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-1734) Un-deprecate the old MapReduce API in the 0.20 branch
[ https://issues.apache.org/jira/browse/MAPREDUCE-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-1734. -- Resolution: Fixed Thanks Harsh Matt. I've committed this to 0.20.205. Un-deprecate the old MapReduce API in the 0.20 branch - Key: MAPREDUCE-1734 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1734 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Todd Lipcon Priority: Blocker Attachments: mapreduce-1734.txt This issue is to un-deprecate the old MapReduce API (in o.a.h.mapred) in the next 0.20 release, as discussed at http://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg01833.html -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2958) mapred-default.xml not merged from mr279
[ https://issues.apache.org/jira/browse/MAPREDUCE-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2958. -- Resolution: Fixed Assignee: Arun C Murthy (was: Mahadev konar) Thanks Thomas, I fixed it by doing an 'svn copy' to both trunk and branch-0.23. mapred-default.xml not merged from mr279 Key: MAPREDUCE-2958 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2958 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Fix For: 0.23.0 I have been running wordcount out of the 23 examples jar. It says it succeeds but doesn't actually output a file. hadoop jar examples/hadoop-mapreduce-0.23.0-SNAPSHOT/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output2 input file is really basic: fdksajl dlkfsajlfljda;j kldfsjallj test one two test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2941) In secure mode RM WebUI shows wrong user for application
In secure mode RM WebUI shows wrong user for application Key: MAPREDUCE-2941 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2941 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Arun C Murthy Priority: Blocker Fix For: 0.23.0 In secure mode RM WebUI shows wrong user for application (mapred) although the RM logs show the right user (me). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2938) Missing log stmt for app submission fail CS
Missing log stmt for app submission fail CS --- Key: MAPREDUCE-2938 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2938 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Trivial Fix For: 0.23.0 Missing log stmt for app submission fail CS -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2191) Findbugs reports 13 warnings on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2191. -- Resolution: Duplicate Duplicate of MAPREDUCE-2193 (which has a patch, albeit stale). Findbugs reports 13 warnings on trunk - Key: MAPREDUCE-2191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2191 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.22.0 Reporter: Eli Collins Attachments: hadoop-findbugs-report.html Findbugs reports 13 warnings on trunk: Warning Type Number Bad practice Warnings 1 Correctness Warnings 5 Multithreaded correctness Warnings6 Performance Warnings 1 Total 13 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2933) Change allocate call to return ContainerStatus for completed containers rather than Container
Change allocate call to return ContainerStatus for completed containers rather than Container -- Key: MAPREDUCE-2933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2933 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Arun C Murthy Change allocate call to return ContainerStatus for completed containers rather than Container, we should do this all the way from the NodeManager too. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2895) Merge AllocateResponse and AMResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2895. -- Resolution: Won't Fix The reason we want AMResponse is to be able to store it to track in on the RM, thus we need to keep them separate. Merge AllocateResponse and AMResponse - Key: MAPREDUCE-2895 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2895 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Fix For: 0.23.0 We need to merge AllocateResponse and AMResponse, having them separate serves no purpose. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2900) Replace ContainerId, Resource in ContainerLaunchContext with Container
[ https://issues.apache.org/jira/browse/MAPREDUCE-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2900. -- Resolution: Won't Fix Wontfix per Vinod's comment. Replace ContainerId, Resource in ContainerLaunchContext with Container -- Key: MAPREDUCE-2900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2900 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Arun C Murthy Fix For: 0.23.0 Replace ContainerId, Resource in ContainerLaunchContext with Container -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2894) Improvements to YARN apis
Improvements to YARN apis - Key: MAPREDUCE-2894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2894 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Fix For: 0.23.0 Ticket to track improvements to YARN apis. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2895) Merge AllocateResponse and AMResponse
Merge AllocateResponse and AMResponse - Key: MAPREDUCE-2895 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2895 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Fix For: 0.23.0 We need to merge AllocateResponse and AMResponse, having them separate serves no purpose. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
