[jira] [Created] (MAPREDUCE-5896) Allow InputSplits to indicate which locations have the block cached in memory
Sandy Ryza created MAPREDUCE-5896: - Summary: Allow InputSplits to indicate which locations have the block cached in memory Key: MAPREDUCE-5896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5896 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5763) Warn message about httpshuffle in NM logs
Sandy Ryza created MAPREDUCE-5763:
-
Summary: Warn message about httpshuffle in NM logs
Key: MAPREDUCE-5763
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5763
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Sandy Ryza
Assignee: Naren Koneru
{code}
2014-02-20 12:08:45,141 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The
Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class
class org.apache.hadoop.mapred.ShuffleHandler which has a name of
'httpshuffle'. Because these are not the same tools trying to send ServiceData
and read Service Meta Data may have issues unless the refer to the name in the
config.
2014-02-20 12:08:45,142 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding
auxiliary service httpshuffle, mapreduce_shuffle
{code}
I'm seeing this in my NodeManager logs, even though things work fine. A WARN
is being caused by some sort of mismatch between the name of the service (in
terms of org.apache.hadoop.service.Service.getName()) and the name of the
auxiliary service.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5759) Remove unnecessary conf load in Limits
Sandy Ryza created MAPREDUCE-5759: - Summary: Remove unnecessary conf load in Limits Key: MAPREDUCE-5759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5759 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sandy Ryza Assignee: Sandy Ryza This is a continuation if MAPREDUCE-5487. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-5745) thread may hang forever, even after it receives all the expected data
[ https://issues.apache.org/jira/browse/MAPREDUCE-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5745. --- Resolution: Invalid thread may hang forever, even after it receives all the expected data - Key: MAPREDUCE-5745 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5745 Project: Hadoop Map/Reduce Issue Type: Wish Reporter: Jinfeng Ni Priority: Trivial Please discard this JIRA issue (I should open it under a different project). Tried to cancel this issue, but could not find a way to do so. Sorry about this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5732) Report proper queue when job has been automatically placed
Sandy Ryza created MAPREDUCE-5732: - Summary: Report proper queue when job has been automatically placed Key: MAPREDUCE-5732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5732 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza Some schedulers, such as the Fair Scheduler, provide the ability to automatically place an application into a queue based on attributes such as the user and group of the submitter. In these cases, the JobHistoryServer and AM web UI report the requested queue, not the queue that the app is actually running in. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5725) TestNetworkedJob relies on the Capacity Scheduler
Sandy Ryza created MAPREDUCE-5725: - Summary: TestNetworkedJob relies on the Capacity Scheduler Key: MAPREDUCE-5725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5725 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza We should either make this explicit or make it scheduler-agnostic. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-5712) Backport Fair Scheduler pool placement by secondary group
[ https://issues.apache.org/jira/browse/MAPREDUCE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5712. --- Resolution: Fixed Assignee: Ted Malaska Hadoop Flags: Reviewed I just committed this to branch-1 Backport Fair Scheduler pool placement by secondary group - Key: MAPREDUCE-5712 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5712 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 1.3.0 Attachments: MAPREDUCE-5712 YARN-1423 introduced a quue police that support selecting a queue if a secondary group was found in the defined queues. This functionality would be useful and minimally invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5651. --- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I just committed this to branch-1. Thanks Ted! Backport Fair Scheduler queue placement policies to branch-1 Key: MAPREDUCE-5651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Ted Malaska Fix For: 1.3.0 Attachments: MAPREDUCE-5651.2.patch, MAPREDUCE-5651.3.patch, MAPREDUCE-5651.4.patch, MAPREDUCE-5651.5.patch, MAPREDUCE-5651.patch YARN-1392 introduced general policies for assigning applications to queues in the YARN fair scheduler. This functionality would be useful and minimally invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (MAPREDUCE-5665) Add audience annotations to MiniMRYarnCluster and MiniMRCluster
Sandy Ryza created MAPREDUCE-5665: - Summary: Add audience annotations to MiniMRYarnCluster and MiniMRCluster Key: MAPREDUCE-5665 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5665 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.2.0 Reporter: Sandy Ryza We should make it clear whether these are public interfaces. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5651) Backport Fair Scheduler queue placement policies to branch-1
Sandy Ryza created MAPREDUCE-5651: - Summary: Backport Fair Scheduler queue placement policies to branch-1 Key: MAPREDUCE-5651 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5651 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza YARN-1392 introduced general policies for assigning applications to queues in the YARN fair scheduler. This functionality would be useful and minimally invasive in MR1 as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5619) Separate out configuration loading from QueueManager in the Fair Scheduler
Sandy Ryza created MAPREDUCE-5619: - Summary: Separate out configuration loading from QueueManager in the Fair Scheduler Key: MAPREDUCE-5619 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5619 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5612) Document TaskAttemptCompletionStatuses
Sandy Ryza created MAPREDUCE-5612: - Summary: Document TaskAttemptCompletionStatuses Key: MAPREDUCE-5612 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5612 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Priority: Minor What's the difference between FAILED and TIPFAILED? What is OBSOLETE? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5608) Replace and deprecate mapred.tasktracker.indexcache.mb
Sandy Ryza created MAPREDUCE-5608: - Summary: Replace and deprecate mapred.tasktracker.indexcache.mb Key: MAPREDUCE-5608 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5608 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza In MR2 mapred.tasktracker.indexcache.mb still works for configuring the size of the shuffle service index cache. As the tasktracker no longer exists, we should replace this with something like mapreduce.shuffle.indexcache.mb. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5596) Allow configuring the number of threads used to serve shuffle connections
Sandy Ryza created MAPREDUCE-5596: - Summary: Allow configuring the number of threads used to serve shuffle connections Key: MAPREDUCE-5596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5596 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza MR1 had mapreduce.tasktracker.http.threads. MR2 always uses the Netty default 2 * Runtime.availableProcessors(). We should make this configurable. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5592) Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1
Sandy Ryza created MAPREDUCE-5592: - Summary: Backport MAPREDUCE-1119 (stack traces on task timeout) in branch-1 Key: MAPREDUCE-5592 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5592 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task-controller Reporter: Sandy Ryza Assignee: Sandy Ryza MAPREDUCE-1119 dumps stack traces on a task timeout, making it easier this difficult case easier to debug. This made it into 0.21, but never into branch-1, and the backport very very dirty. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5578) Miscellaneous Fair Scheduler speedups
Sandy Ryza created MAPREDUCE-5578: - Summary: Miscellaneous Fair Scheduler speedups Key: MAPREDUCE-5578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5578 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Reporter: Sandy Ryza Assignee: Sandy Ryza I ran the Fair Scheduler's core scheduling loop through a profiler to and identified a bunch of minimally invasive changes that can shave off a few milliseconds. The main one is demoting a couple INFO log messages to DEBUG, which brought my benchmark down from 16000 ms to 6000. A few others (which had way less of an impact) were * Most of the time in comparisons was being spent in Math.signum. I switched this to direct ifs and elses and it halved the percent of time spent in comparisons. * I removed some unnecessary instantiations of Resource objects * I made it so that queues' usage wasn't calculated from the applications up each time getResourceUsage was called. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5575) History files deleted from the intermediate directory never get removed from the JobListCache
Sandy Ryza created MAPREDUCE-5575: - Summary: History files deleted from the intermediate directory never get removed from the JobListCache Key: MAPREDUCE-5575 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5575 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.2.0 Reporter: Sandy Ryza The JobHistoryServer periodically scans through the intermediate directory. It adds all files to the JobListCache. It deletes job files that are older than the max age and moves all other files to the done directory. Later, when files in the done directory become too old, they're deleted from the JobListCache. Jobs that were deleted in the intermediate directory (and thus never moved to the done directory) end up in the JobListCache but can never be deleted from it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
Sandy Ryza created MAPREDUCE-5577: - Summary: Allow querying the JobHistoryServer by job arrival time Key: MAPREDUCE-5577 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Sandy Ryza Assignee: Sandy Ryza The JobHistoryServer REST APIs currently allow querying by job submit time and finish time. However, jobs don't necessarily arrive in order of their finish time, meaning that a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. The arrival time of a job would be defined as the time that it lands in the done directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5544) JobClient#getJob loads job conf twice
Sandy Ryza created MAPREDUCE-5544: - Summary: JobClient#getJob loads job conf twice Key: MAPREDUCE-5544 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5544 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Calling JobClient#getJob causes the job conf file to be loaded twice, once in the constructor of JobClient.NetworkedJob and once in Cluster#getJob. We should remove the former. MAPREDUCE-5001 was meant to fix a race that was causing problems in Hive tests, but the problem persists because it only fixed one of the places where the job conf file is loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5527) Add CONTAINERS_MILLIS_MAPS|REDUCES counters
Sandy Ryza created MAPREDUCE-5527: - Summary: Add CONTAINERS_MILLIS_MAPS|REDUCES counters Key: MAPREDUCE-5527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5527 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza It would be helpful to have a counters which report the total wallclock time spent in all map/reduce tasks. This is what SLOTS_MILLIS_MAPS usually did in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
Sandy Ryza created MAPREDUCE-5487: - Summary: In task processes, JobConf is unnecessarily loaded again in Limits Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5484) YarnChild unnecessarily loads job conf twice
Sandy Ryza created MAPREDUCE-5484: - Summary: YarnChild unnecessarily loads job conf twice Key: MAPREDUCE-5484 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5484 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza In MR task processes, a JobConf is instantiated with the same job.xml twice, once at the beginning of main() and once in configureTask. IIUC, the second instantiation is not necessary. These take time reading from disk and parsing XML. Removing the second instantiation shaved a second off the average map task time in a 1,000-map sleep job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5478) TeraInputFormat unnecessarily defines its own FileSplit subclass
Sandy Ryza created MAPREDUCE-5478: - Summary: TeraInputFormat unnecessarily defines its own FileSplit subclass Key: MAPREDUCE-5478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5478 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor TeraInputFormat defines its own TeraFileSplit subclass of FileSplit that adds a locations field, which is already included in FileSplit. This is causing MR2 TeraSort to fail on MR1, which, for a System.arraycopy, requires splits to be of the FileSplit class. While nobody is promising that everything that runs on MR2 should run on MR1, fixing this would be easy and make it possible to compare MR2 TeraSort performance between MR1 and MR2. We should just get rid of TeraFileSplit and use FileSplit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5462) In map-side sort, swap entire meta entries instead of indexes for better cache performance
Sandy Ryza created MAPREDUCE-5462: - Summary: In map-side sort, swap entire meta entries instead of indexes for better cache performance Key: MAPREDUCE-5462 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5462 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5463) Deprecate SLOTS_MILLIS counters
Sandy Ryza created MAPREDUCE-5463: - Summary: Deprecate SLOTS_MILLIS counters Key: MAPREDUCE-5463 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5463 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza As discussed in MAPREDUCE-5311, the SLOTS_MILLIS_MAPS and SLOTS_MILLIS_REDUCES counters don't really make sense in MR2, and should be deprecated so that they can eventually be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5464) Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter
Sandy Ryza created MAPREDUCE-5464: - Summary: Add MEM_MILLIS_MAPS and MEM_MILLIS_REDUCES counter Key: MAPREDUCE-5464 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5464 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Per discussion on MAPREDUCE-5311, it would be good to have analogs for SLOTS_MILLIS that better fit the MR2 resource model. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5457) Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators
Sandy Ryza created MAPREDUCE-5457: - Summary: Add a KeyOnlyTextOutputFormat to enable streaming write out text files without separators Key: MAPREDUCE-5457 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5457 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: Sandy Ryza MR jobs sometimes want to just output lines of text, not key/value pairs. TextOutputFormat handles this by, if a null value is given, outputting only the key with no separator. Streaming jobs are unable to take advantage of this, because they can't output null values. A text output format that ignores values and only outputs keys would allow streaming jobs to output lines of text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5450) Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5450. --- Resolution: Fixed Hadoop Flags: Reviewed Unnecessary Configuration instantiation in IFileInputStream slows down merge - Port to branch-1 --- Key: MAPREDUCE-5450 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5450 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Blocker Fix For: 1.3.0 Attachments: MAPREDUCE-5450-1.1.0.txt, mapreduce-5450.txt We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) -
[jira] [Reopened] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5311: --- Remove slot millis computation logic and deprecate counter constants Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Attachments: MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5059: --- Job overview shows average merge time larger than for any reduce attempt Key: MAPREDUCE-5059 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Reporter: Jason Lowe Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta, 0.23.8 Attachments: MAPREDUCE-5059-20130325.patch, MAPREDUCE-5059-20130410.patch When looking at a job overview page on the history server, the Average Merge Time is often reported with a value that is far larger than the Elapsed Merge Time shown for any reduce task attempt. The job overview page calculates the merge time as the time delta between the sort finishing and the job launching while the attempts page calculates it as the time delta between the sort finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5059) Job overview shows average merge time larger than for any reduce attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5059. --- Resolution: Fixed Job overview shows average merge time larger than for any reduce attempt Key: MAPREDUCE-5059 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5059 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Reporter: Jason Lowe Assignee: Omkar Vinit Joshi Fix For: 2.1.0-beta, 0.23.8 Attachments: MAPREDUCE-5059-20130325.patch, MAPREDUCE-5059-20130410.patch When looking at a job overview page on the history server, the Average Merge Time is often reported with a value that is far larger than the Elapsed Merge Time shown for any reduce task attempt. The job overview page calculates the merge time as the time delta between the sort finishing and the job launching while the attempts page calculates it as the time delta between the sort finishing and the shuffle finishing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5420) Remove mapreduce.task.tmp.dir from mapred-default.xml
Sandy Ryza created MAPREDUCE-5420: - Summary: Remove mapreduce.task.tmp.dir from mapred-default.xml Key: MAPREDUCE-5420 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5420 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza mapreduce.task.tmp.dir no longer has any effect, so it should no longer be documented in mapred-default. (There is no YARN equivalent for the property. It now is just always ./tmp). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5403) Get rid of yarn.application.classpath
Sandy Ryza created MAPREDUCE-5403: - Summary: Get rid of yarn.application.classpath Key: MAPREDUCE-5403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza yarn.application.classpath is a confusing property because it is used by MapReduce and not YARN, and MapReduce already has mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5382) LocalJobRunner should use default FS for system and staging dirs by default
Sandy Ryza created MAPREDUCE-5382: - Summary: LocalJobRunner should use default FS for system and staging dirs by default Key: MAPREDUCE-5382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5382 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2, 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza For local jobs, staging dirs and system dirs are currently required to be placed on the local FS. I am continually bitten by permissions errors when I set mapreduce.jobtracker.staging.root.dir to /user, even when the default FS is still HDFS. I think using a different FS for staging than the default FS is confusing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5383) Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir
Sandy Ryza created MAPREDUCE-5383: - Summary: Deprecate to mapreduce.jobtracker.staging.root.dir to yarn.app.mapreduce.am.staging-dir Key: MAPREDUCE-5383 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5383 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza This will allow configurations that had previously set mapreduce.jobtracker.staging.root.dir should be able to more easily transition to MR2, as well as make it clear that these properties refer to the same thing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5379) Include FS delegation token ID in job conf
Sandy Ryza created MAPREDUCE-5379: - Summary: Include FS delegation token ID in job conf Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission, security Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5372) ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2
Sandy Ryza created MAPREDUCE-5372: - Summary: ControlledJob#getMapredJobID capitalization is inconsistent between MR1 and MR2 Key: MAPREDUCE-5372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5372 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Sandy Ryza In MR2, the 'd' in Id is lowercase, but in MR1, it is capitalized. While ControlledJob is marked as Evolving, there is no reason to be inconsistent here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5363) Fix doc and spelling for TaskCompletionEvent#getTaskStatus and getStatus
Sandy Ryza created MAPREDUCE-5363:
-
Summary: Fix doc and spelling for
TaskCompletionEvent#getTaskStatus and getStatus
Key: MAPREDUCE-5363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 1.1.2, 2.1.0-beta
Reporter: Sandy Ryza
The doc for TaskCompletionEvent#get(Task)Status in both MR1 and MR2 is
{code}
Returns enum Status.SUCESS or Status.FAILURE.
@return task tracker status
{code}
The actual values that the Status enum can take are
FAILED, KILLED, SUCCEEDED, OBSOLETE, TIPFAILED
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5365) Set mapreduce.job.classpath to true by default
Sandy Ryza created MAPREDUCE-5365: - Summary: Set mapreduce.job.classpath to true by default Key: MAPREDUCE-5365 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5365 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza MAPREDUCE-1700 introduced the mapreduce.job.classpath option, which uses a custom classloader to separate system classes from user classes. It seems like there are only rare cases when a user would not want this on, and that it should enabled by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5367) Local jobs all use same local working directory
Sandy Ryza created MAPREDUCE-5367: - Summary: Local jobs all use same local working directory Key: MAPREDUCE-5367 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5367 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza This means that local jobs, even in different JVMs, can't run concurrently because they might delete each other's files during work directory setup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5350) Expose Fair Scheduler-specific queue metrics
Sandy Ryza created MAPREDUCE-5350: - Summary: Expose Fair Scheduler-specific queue metrics Key: MAPREDUCE-5350 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5350 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When the Fair Scheduler is enabled, QueueMetrics should include fair share, minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5351) JobTracker memory leak caused by CleanupQueue reopening FileSystem
Sandy Ryza created MAPREDUCE-5351: - Summary: JobTracker memory leak caused by CleanupQueue reopening FileSystem Key: MAPREDUCE-5351 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5351 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Critical When a job is completed, closeAllForUGI is called to close all the cached FileSystems in the FileSystem cache. However, the CleanupQueue may run after this occurs and call FileSystem.get() to delete the staging directory, adding a FileSystem to the cache that will never be closed. People on the user-list have reported this causing their JobTrackers to OOME every two weeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5338) Bring back mapred.child.ulimit
Sandy Ryza created MAPREDUCE-5338: - Summary: Bring back mapred.child.ulimit Key: MAPREDUCE-5338 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5338 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza In MR1, a ulimit could be set for MapReduce child processes. For parity, this would be good to have in MR2 as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5321) Enable better parallelism in the Fair Scheduler
Sandy Ryza created MAPREDUCE-5321: - Summary: Enable better parallelism in the Fair Scheduler Key: MAPREDUCE-5321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5321 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza Currently, the Fair Scheduler is locked on pretty much every operation, node updates, application additions and removals, every time the update thread runs, and every time the RM queries it for information. Most of this locking is unnecessary, especially as only the core scheduling operations like application additions, removals, and node updates need a consistent view of scheduler state. We can probably increase parallelism by using concurrent data structures when applicable, as well as keeping a slightly stale view to serve via the RM APIs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest
[ https://issues.apache.org/jira/browse/MAPREDUCE-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5252. --- Resolution: Not A Problem This was fixed in YARN-326 Fair scheduler should use SchedulerUtils.normalizeRequest - Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor The capacity scheduler and the fifo scheduler use the same normalizeRequest in SchedulerUtils. The fair scheduler has its own version of this method that does exactly the same thing. It should use the common one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5182. --- Resolution: Won't Fix LineRecordReader#getProgress throwing IOException breaks compatibility -- Key: MAPREDUCE-5182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5182 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2, 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza This has been in trunk for a while (since MAPREDUCE-773), but was only introduced into branch-1 in July. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5302) NodeManager throws AvroRuntimeException on failed start
Sandy Ryza created MAPREDUCE-5302: - Summary: NodeManager throws AvroRuntimeException on failed start Key: MAPREDUCE-5302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5302 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza NodeManager wraps exceptions that occur in its start method in AvroRuntimeExceptions, even though it doesn't use Avro anywhere else. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
Sandy Ryza created MAPREDUCE-5283: - Summary: Over 10 different tests have near identical implementations of AppContext Key: MAPREDUCE-5283 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, test Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go into nearly every test file for MR web services to make sure their TestAppContext implements it. I propose having a common implementation of AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
[ https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5036: --- Default shuffle handler port should not be 8080 --- Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.5-beta Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036.patch The shuffle handler port (mapreduce.shuffle.port) defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest
Sandy Ryza created MAPREDUCE-5252: - Summary: Fair scheduler should use SchedulerUtils.normalizeRequest Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor The capacity scheduler and the fifo scheduler use the same normalizeRequest in SchedulerUtils. The fair scheduler has its own version of this method that does exactly the same thing. It should use the common one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5238) TestDistCacheEmulation.testGenerateDistCacheData is failing in trunk
Sandy Ryza created MAPREDUCE-5238:
-
Summary: TestDistCacheEmulation.testGenerateDistCacheData is
failing in trunk
Key: MAPREDUCE-5238
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5238
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
{noformat
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.794 sec
FAILURE!
testGenerateDistCacheData(org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation)
Time elapsed: 16767 sec FAILURE!
java.lang.AssertionError: Wrong permissions for distributed cache file
/user/sandy/testSetupGenerateDistCacheData/distributedCache/26046e44bfac7cec1afce2cef15ee281
expected:rw-r--r-- but was:rwxr-xr-x
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheFiles(TestDistCacheEmulation.java:136)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.validateDistCacheData(TestDistCacheEmulation.java:109)
at
org.apache.hadoop.mapred.gridmix.TestDistCacheEmulation.testGenerateDistCacheData(TestDistCacheEmulation.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5219) JobStatus#getJobPriority changed to JobStatus#getPriority in MR2
Sandy Ryza created MAPREDUCE-5219: - Summary: JobStatus#getJobPriority changed to JobStatus#getPriority in MR2 Key: MAPREDUCE-5219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza We should change it back for compatibility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5220) Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2
Sandy Ryza created MAPREDUCE-5220: - Summary: Setter methods in TaskCompletionEvent are public in MR1 and protected in MR2 Key: MAPREDUCE-5220 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5220 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: client Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3946) If a resource requirement is higher than available on any node, job should fail early
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-3946. --- Resolution: Duplicate If a resource requirement is higher than available on any node, job should fail early - Key: MAPREDUCE-3946 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.24.0, 0.23.2 Reporter: Todd Lipcon If you configure the NMs to have 1GB of RAM each, and then try to submit a job which has an AM resource requirement of 1.5GB, the job will neither run nor fail. Instead, it will slowly sop of all of the resources in the cluster as reservations despite the fact that it will never be able to schedule something. Instead, it should fail early indicating that the required memory allocation is infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5134) Default settings cause LocalJobRunner to OOME
[ https://issues.apache.org/jira/browse/MAPREDUCE-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5134. --- Resolution: Not A Problem Default settings cause LocalJobRunner to OOME - Key: MAPREDUCE-5134 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5134 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza If I run a job using the local job runner with vanilla settings, I get an out of memory error. This seems to be because the default client memory maximum is 128 MB, and the default io.sort.mb is 100 MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5185) When log aggregation not enabled, message should point to NM HTTP port, not IPC port
Sandy Ryza created MAPREDUCE-5185: - Summary: When log aggregation not enabled, message should point to NM HTTP port, not IPC port Key: MAPREDUCE-5185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5185 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When I try to get a container's logs in the JHS without log aggregation enabled, I get a message that looks like this: Aggregation is not enabled. Try the nodemanager at sandy-ThinkPad-T530:33224 This could be a lot more helpful by actually pointing the URL that would show the container logs on the NM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility
Sandy Ryza created MAPREDUCE-5182: - Summary: LineRecordReader#getProgress throwing IOException breaks compatibility Key: MAPREDUCE-5182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5182 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2, 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza This has been in trunk for a while (since MAPREDUCE-773), but was only introduced into branch-1 in July. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
Sandy Ryza created MAPREDUCE-5183: - Summary: In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign Key: MAPREDUCE-5183 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, tasktracker Affects Versions: 1.1.2 Reporter: Sandy Ryza Priority: Minor This makes looking at progress in the logs unnecessarily confusing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5171) Expose info about blacklisted nodes from MR AM REST API
Sandy Ryza created MAPREDUCE-5171: - Summary: Expose info about blacklisted nodes from MR AM REST API Key: MAPREDUCE-5171 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5171 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5172) Expose number of non-local maps in MR app metrics
Sandy Ryza created MAPREDUCE-5172: - Summary: Expose number of non-local maps in MR app metrics Key: MAPREDUCE-5172 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5172 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5173) Need to specify master principal run local job in a secure setup
Sandy Ryza created MAPREDUCE-5173: - Summary: Need to specify master principal run local job in a secure setup Key: MAPREDUCE-5173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5173 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, security Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When security is turned on, running a job involves asking the filesystems it uses for delegation tokens. Currently, this fails if the kerberos master principal is not set, even if the job is run against a filesystem that doesn't use delegation tokens -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3603) Add Web UI to MR2 Fair Scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-3603. --- Resolution: Duplicate Assignee: (was: Patrick Wendell) Add Web UI to MR2 Fair Scheduler Key: MAPREDUCE-3603 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3603 Project: Hadoop Map/Reduce Issue Type: New Feature Components: scheduler Reporter: Patrick Wendell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5154) staging directory deletion fails because delegation tokens have been cancelled
Sandy Ryza created MAPREDUCE-5154: - Summary: staging directory deletion fails because delegation tokens have been cancelled Key: MAPREDUCE-5154 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5154 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza In a secure setup, the jobtracker needs the job's delegation tokens to delete the staging directory. MAPREDUCE-4850 made it so that job cleanup staging directory deletion occurs asynchronously, so that it could order it with system directory deletion. This introduced the issue that a job's delegation tokens could be cancelled before the cleanup thread got around to deleting it, causing the deletion to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4676) Add test for job history cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-4676: --- Add test for job history cleaner Key: MAPREDUCE-4676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4676 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3, 2.0.1-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4676.patch, MAPREDUCE-4676-trunk.patch Add a test to TestJobHistory that verifies that the HistoryCleaner cleans up the job history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5133) TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race
Sandy Ryza created MAPREDUCE-5133: - Summary: TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race Key: MAPREDUCE-5133 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5133 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza At the end of TestSubmitJob.testSecureJobExecution, the test waits for the job to be done and then asserts that the job submission directory has been deleted. The directory is deleted by an asynchronous cleanup thread, so the test can hit the assert before the deletion is run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5134) Default settings cause LocalJobRunner to OOME
Sandy Ryza created MAPREDUCE-5134: - Summary: Default settings cause LocalJobRunner to OOME Key: MAPREDUCE-5134 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5134 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza If I run a job using the local job runner with vanilla settings, I get an out of memory error. This seems to be because the default client memory maximum is 128 MB, and the default io.sort.mb is 100 MB. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
Sandy Ryza created MAPREDUCE-5128: - Summary: mapred-default.xml is missing a bunch of history server configs Key: MAPREDUCE-5128 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza mapred-default.xml is missing many configs that work for the job history server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
Sandy Ryza created MAPREDUCE-5130: - Summary: Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza I came across that mapreduce.map.child.java.opts and mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5120) Allow app master to use tracing async dispatcher
Sandy Ryza created MAPREDUCE-5120: - Summary: Allow app master to use tracing async dispatcher Key: MAPREDUCE-5120 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5120 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza YARN-366 proposes an option to add traces to events so that exceptions could report an events lineage. This JIRA would add a mapreduce config option that would allow the MR app master to use the tracing async dispatcher as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5111) Unnecessary RM address DNS lookup can cause unnecessary slowness on every JHS page load
Sandy Ryza created MAPREDUCE-5111: - Summary: Unnecessary RM address DNS lookup can cause unnecessary slowness on every JHS page load Key: MAPREDUCE-5111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5111 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When I run the job history server locally, every page load takes in the 10s of seconds. I profiled the process and discovered that all the extra time was spent inside YarnConfiguration#getRMWebAppURL, trying to resolve 0.0.0.0 to a hostname. When I changed my yarn.resourcemanager.address to localhost, the page load times decreased drastically. There's no that we need to perform this resolution on every page load. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5038) old API CombineFileInputFormat missing fixes that are in new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-5038: --- old API CombineFileInputFormat missing fixes that are in new API - Key: MAPREDUCE-5038 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0 Attachments: MAPREDUCE-5038-1.patch, MAPREDUCE-5038.patch The following changes patched the CombineFileInputFormat in mapreduce, but neglected the one in mapred MAPREDUCE-1597 enabled the CombineFileInputFormat to work on splittable files MAPREDUCE-2021 solved returning duplicate hostnames in split locations MAPREDUCE-1806 CombineFileInputFormat does not work with paths not on default FS In trunk this is not an issue as the one in mapred extends the one in mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5076) CombineFileInputFormat with maxSplitSize can omit data
Sandy Ryza created MAPREDUCE-5076: - Summary: CombineFileInputFormat with maxSplitSize can omit data Key: MAPREDUCE-5076 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5076 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sandy Ryza Assignee: Sandy Ryza I ran a local job with CombineFileInputFormat using an 80 MB file and a max split size of 32 MB (the default local FS block size). The job ran with two splits of 32 MB, and the last 16 MB were just omitted. This appears to be caused by a subtle bug in getMoreSplits, in which the code that generates the splits from the blocks expects the 16 MB block to be at the end of the block list. But the code that generates the blocks does not respect this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5070) TestClusterStatus.testClusterMetrics fails on JDK7
Sandy Ryza created MAPREDUCE-5070: - Summary: TestClusterStatus.testClusterMetrics fails on JDK7 Key: MAPREDUCE-5070 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5070 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza TestClusterStatus is sensitive to the order that the tests are run in. If testReservedSlots is called before testClusterMetrics, testClusterMetrics will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5072) TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7
Sandy Ryza created MAPREDUCE-5072: - Summary: TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7 Key: MAPREDUCE-5072 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5072 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0 TestDelegationTokenRenewal.testDTRenewal fails in MR1 for the reasons that TestDelegationTokenRenewer.testDTRenewal fails described in YARN-31. The fix is the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5073) TestJobStatusPersistency.testPersistency fails on JDK7
Sandy Ryza created MAPREDUCE-5073: - Summary: TestJobStatusPersistency.testPersistency fails on JDK7 Key: MAPREDUCE-5073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5073 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.1.2 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0 TestJobStatusPersistency is sensitive to the order that the tests are run in. If testLocalPersistency runs before testPersistency, testPersistency will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5047) keep.failed.task.files=true causes job failure on secure clusters
Sandy Ryza created MAPREDUCE-5047: - Summary: keep.failed.task.files=true causes job failure on secure clusters Key: MAPREDUCE-5047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5047 Project: Hadoop Map/Reduce Issue Type: Bug Components: task, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza To support IsolationRunner, split info is written to local directories. This occurs inside MapTask#localizeConfiguration, which is called both tasktracker and by the child JVM. On a secure cluster, the tasktacker's attempt to write it fails, because the tasktracker does not have permission to write to the user's directory. It is likely that the call to localizeConfiguration in the tasktracker can be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5049) CombineFileInputFormat counts all compressed files non-splitable
Sandy Ryza created MAPREDUCE-5049: - Summary: CombineFileInputFormat counts all compressed files non-splitable Key: MAPREDUCE-5049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5049 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza In branch-1, CombineFileInputFormat doesn't take SplittableCompressionCodec into account and thinks that all compressible input files aren't splittable. This is a regression from when handling for non-splitable compression codecs was originally added in MAPREDUCE-1597, and seems to have somehow gotten in when the code was pulled from 0.22 to branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5038) mapred CombineFileInputFormat does not work on non-splittable files
Sandy Ryza created MAPREDUCE-5038: - Summary: mapred CombineFileInputFormat does not work on non-splittable files Key: MAPREDUCE-5038 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5038 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza MAPREDUCE-1597 enabled the CombineFileInputFormat in mapreduce to work on splittable files, but neglected to consider the one in mapred. In trunk this is not an issue as the one in mapred extends the one in mapreduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5036) Default shuffle handler port should not be 8080
Sandy Ryza created MAPREDUCE-5036: - Summary: Default shuffle handler port should not be 8080 Key: MAPREDUCE-5036 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza The shuffle handler port defaults to 8080. This is a pretty common port for web services, and is likely to cause unnecessary port conflicts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4933) MR1 final merge asks for length of file it just wrote before flushing it
[
https://issues.apache.org/jira/browse/MAPREDUCE-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza resolved MAPREDUCE-4933.
---
Resolution: Fixed
MR1 final merge asks for length of file it just wrote before flushing it
Key: MAPREDUCE-4933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4933
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, task
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4933-branch-1.patch
createKVIterator in ReduceTask contains the following code:
{code}
try {
Merger.writeFile(rIter, writer, reporter, job);
addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath));
} catch (Exception e) {
if (null != outputPath) {
fs.delete(outputPath, true);
}
throw new IOException(Final merge failed, e);
} finally {
if (null != writer) {
writer.close();
}
}
{code}
Merger#writeFile() does not close the file after writing it, so when
fs.getFileStatus() is called on it, it may not return the correct length.
This causes bad accounting further down the line, which can lead to map
output data being lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5021) Add an addDirectoryToClassPath method DistributedCache
Sandy Ryza created MAPREDUCE-5021: - Summary: Add an addDirectoryToClassPath method DistributedCache Key: MAPREDUCE-5021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5021 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, distributed-cache Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza As adding a directory of jars to the class path is a common use for the distributed cache it would be easier on API consumers if they were able to call a method that would add all the the files in a directory for them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5013) JobStatus compatibility: MR2 missing constructors from MR1
Sandy Ryza created MAPREDUCE-5013: - Summary: JobStatus compatibility: MR2 missing constructors from MR1 Key: MAPREDUCE-5013 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5013 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza JobStatus is missing the following constructors in MR2 that were present in MR1 public org.apache.hadoop.mapred.JobStatus(org.apache.hadoop.mapred.JobID, float, float, float, int); public org.apache.hadoop.mapred.JobStatus(org.apache.hadoop.mapred.JobID, float, float, int); public org.apache.hadoop.mapred.JobStatus(org.apache.hadoop.mapred.JobID, float, float, float, int, org.apache.hadoop.mapred.JobPriority); public org.apache.hadoop.mapred.JobStatus(org.apache.hadoop.mapred.JobID, float, float, float, float, int, org.apache.hadoop.mapred.JobPriority); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5008) Merger progress miscounts with respect to EOF_MARKER
Sandy Ryza created MAPREDUCE-5008: - Summary: Merger progress miscounts with respect to EOF_MARKER Key: MAPREDUCE-5008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5008 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza After MAPREDUCE-2264, a segment's raw data length is calculated without the EOF_MARKER bytes. However, when the merge is counting how many bytes it processed, it includes the marker. This can cause the merge progress to go above 100%. Whether these EOF_MARKER bytes should count should be consistent between the two. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-4994: --- -jt generic command line option does not work - Key: MAPREDUCE-4994 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.4-beta Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ should run a job using the local file system and the local job runner. Instead it tries to connect to a jobtracker. hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ does not use the given host/port This appears to be because Cluster#initialize, which loads the ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4643) Make job-history cleanup-period configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reopened MAPREDUCE-4643: --- Make job-history cleanup-period configurable Key: MAPREDUCE-4643 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1.0.3 Reporter: Karthik Kambatla Assignee: Sandy Ryza Fix For: 1.2.0 Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch-1-4.patch, MAPREDUCE-4643-branch-1-5.patch, MAPREDUCE-4643-branch-1-6.patch, MAPREDUCE-4643-branch-1-addendum.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4997) Deprecate mapreduce.jobtracker.address
Sandy Ryza created MAPREDUCE-4997: - Summary: Deprecate mapreduce.jobtracker.address Key: MAPREDUCE-4997 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4997 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza mapreduce.jobtracker.address currently is not used, but users transitioning from mr1 to mr2 may expect their previous job configs to work, so it should be deprecated in favor of yarn.resourcemanager.address. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4994) Can't submit local job with hadoop jar -jt local
Sandy Ryza created MAPREDUCE-4994: - Summary: Can't submit local job with hadoop jar -jt local Key: MAPREDUCE-4994 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ should run a job using the local file system and the local job runner. Instead it tries to connect to a jobtracker. This appears to be because Cluster#initialize, which loads the ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4922) Request with multiple data local nodes can cause NPE in AppSchedulingInfo
[ https://issues.apache.org/jira/browse/MAPREDUCE-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-4922. --- Resolution: Won't Fix This is the expected behavior Request with multiple data local nodes can cause NPE in AppSchedulingInfo - Key: MAPREDUCE-4922 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4922 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mr-am, mrv2, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza With the way that the schedulers work, each request for a container on a node must consist of 3 ResourceRequests - one on the node, one on the rack, and one with *. AppSchedulingInfo tracks the outstanding requests. When a node is assigned a node-local container, allocateNodeLocal decrements the outstanding requests at each level - node, rack, and *. If the rack requests reach 0, it removes the mapping. A mapreduce task with multiple data local nodes submits multiple container requests, one for each node. It also submits one for each unique rack, and one for *. If there are fewer unique racks than data local nodes, this means that fewer rack-local ResourceRequests will be submitted than node-local ResourceRequests, so the rack-local mapping will be deleted before all the node-local requests are allocated and an NPE will come up the next time a node-local request from that rack is allocated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
Sandy Ryza created MAPREDUCE-4984: - Summary: AM leaves unwanted node/rack requests after container has been assigned Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4984) AM leaves unwanted node/rack requests after container has been assigned
[ https://issues.apache.org/jira/browse/MAPREDUCE-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-4984. --- Resolution: Duplicate AM leaves unwanted node/rack requests after container has been assigned --- Key: MAPREDUCE-4984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4984 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mr-am Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Consider the following situation: node1, node2, and node3 are all on rack1 task1 is submitted with resource requests on node1, node2, rack1, and * task2 is submitted with resource requests on node3, rack1, and * The RM gives a container to the app on node1, on which the AM assigns task1. While node1 is removed from the scheduler's bookkeeping, node2 is not, so its delay scheduling will try as hard to assign a container there is it would to node3. The AM should cancel its request for node2 on its next heartbeat by sending a resource request with # containers = 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4970) Child tasks create security audit log files
Sandy Ryza created MAPREDUCE-4970: - Summary: Child tasks create security audit log files Key: MAPREDUCE-4970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.1 Reporter: Sandy Ryza After HADOOP-8552, MR child tasks will attempt to create security audit log files with their user names. On an insecure cluster, this has no effect, but on a secure cluster, empty log files will be created for tasks with names like SecurityAuth-joeuser.log. I haven't verified whether this occurs in MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4962) jobdetails.jsp uses display name instead of real name to get counters
Sandy Ryza created MAPREDUCE-4962: - Summary: jobdetails.jsp uses display name instead of real name to get counters Key: MAPREDUCE-4962 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4962 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker, mrv1 Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza jobdetails.jsp displays details for a job including its counters. Counters may have different real names and display names, but the display names are used to look the counter values up, so counter values can incorrectly show up as 0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4951) Container preemption interpreted as task failure
Sandy Ryza created MAPREDUCE-4951: - Summary: Container preemption interpreted as task failure Key: MAPREDUCE-4951 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4951 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mr-am, mrv2 Reporter: Sandy Ryza Assignee: Sandy Ryza When YARN reports a completed container to the MR AM, it always interprets it as a failure. This can lead to a job failing because too many of its tasks failed, when in fact they only failed because the scheduler preempted them. MR needs to recognize the special exit code value of -100 and interpret it as a container being killed instead of a container failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4952) FSSchedulerNode is always instantiated with a 0 virtual core capacity
[ https://issues.apache.org/jira/browse/MAPREDUCE-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-4952. --- Resolution: Invalid Sorry, this should be YARN JIRA - filed YARN-350 instead FSSchedulerNode is always instantiated with a 0 virtual core capacity - Key: MAPREDUCE-4952 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4952 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza After YARN-2, FSSchedulerNode was not updated to initialize with the underlying RMNode's CPU capacity, and thus always has 0 virtual cores. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4949) Enable multiple pi jobs to run in parallel
Sandy Ryza created MAPREDUCE-4949: - Summary: Enable multiple pi jobs to run in parallel Key: MAPREDUCE-4949 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4949 Project: Hadoop Map/Reduce Issue Type: Improvement Components: examples Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor Currently the hadoop pi example uses a hardcoded temporary directory to store its inputs and outputs. This makes it so that only one pi job can run at a time, and that if it is cancelled, the temporary directory must be manually deleted. I propose using a temporary directory based on a timestamp and random number to avoid these conflicts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4933) MR1 merger asks for length of file it just wrote before flushing it
Sandy Ryza created MAPREDUCE-4933:
-
Summary: MR1 merger asks for length of file it just wrote before
flushing it
Key: MAPREDUCE-4933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4933
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, task
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
createKVIterator in ReduceTask contains the following code:
{code}
try {
Merger.writeFile(rIter, writer, reporter, job);
addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath));
} catch (Exception e) {
if (null != outputPath) {
fs.delete(outputPath, true);
}
throw new IOException(Final merge failed, e);
} finally {
if (null != writer) {
writer.close();
}
}
{code}
Merger#writeFile() does not close the file after writing it, so when
fs.getFileStatus() is called on it, it may not return the correct length. This
causes bad accounting further down the line, which can lead to map output data
being lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
Sandy Ryza created MAPREDUCE-4929: - Summary: mapreduce.task.timeout is ignored Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Sandy Ryza In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4922) Request with multiple data local nodes can cause NPE in AppSchedulingInfo
Sandy Ryza created MAPREDUCE-4922: - Summary: Request with multiple data local nodes can cause NPE in AppSchedulingInfo Key: MAPREDUCE-4922 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4922 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza With the way that the schedulers work, each request for a container on a node must consist of 3 ResourceRequests - one on the node, one on the rack, and one with *. AppSchedulingInfo tracks the outstanding requests. When a node is assigned a node-local container, allocateNodeLocal decrements the outstanding requests at each level - node, rack, and *. If the rack requests reach 0, it removes the mapping. A mapreduce task with multiple data local nodes submits multiple container requests, one for each node. It also submits one for each unique rack, and one for *. If there are fewer unique racks than data local nodes, this means that fewer rack-local ResourceRequests will be submitted than node-local ResourceRequests, so the rack-local mapping will be deleted before all the node-local requests are allocated and an NPE will come up the next time a node-local request from that rack is allocated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4923) Add toString method to TaggedInputSplit
Sandy Ryza created MAPREDUCE-4923: - Summary: Add toString method to TaggedInputSplit Key: MAPREDUCE-4923 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4923 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2, task Affects Versions: 2.0.2-alpha, 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor Per MAPREDUCE-3678, map task logs now contain information about the input split being processed. Because TaggedInputSplit has no overridden toString method, nothing useful gets printed out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
Sandy Ryza created MAPREDUCE-4907: - Summary: TrackerDistributedCacheManager issues too many getFileStatus calls Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
