[jira] [Updated] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5428: Status: Patch Available (was: Open) Kicking Jenkins, now that HADOOP-9787 is committed. HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[
https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726116#comment-13726116
]
Hadoop QA commented on MAPREDUCE-5428:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12595086/mr-5428-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3923//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3923//console
This message is automatically generated.
HistoryFileManager doesn't stop threads when service is stopped
---
Key: MAPREDUCE-5428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
Attachments: mr-5428-1.patch
HistoryFileManager is a service that starts threads, but it does not override
the serviceStop method to stop the threads when the service is stopped.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5352) Optimize node local splits generated by CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726127#comment-13726127 ] Bikas Saha commented on MAPREDUCE-5352: --- Sounds good. +1. Optimize node local splits generated by CombineFileInputFormat --- Key: MAPREDUCE-5352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5352 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5352.1.txt, MAPREDUCE-5352.2.txt, MAPREDUCE-5352.3.txt, MAPREDUCE-5352.4.txt CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse. Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1176) Contribution: FixedLengthInputFormat and FixedLengthRecordReader
[ https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1176: - Assignee: (was: Chris Douglas) Contribution: FixedLengthInputFormat and FixedLengthRecordReader Key: MAPREDUCE-1176 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 0.20.1, 0.20.2 Environment: Any Reporter: BitsOfInfo Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch, MAPREDUCE-1176-v3.patch, MAPREDUCE-1176-v4.patch Hello, I would like to contribute the following two classes for incorporation into the mapreduce.lib.input package. These two classes can be used when you need to read data from files containing fixed length (fixed width) records. Such files have no CR/LF (or any combination thereof), no delimiters etc, but each record is a fixed length, and extra data is padded with spaces. The data is one gigantic line within a file. Provided are two classes first is the FixedLengthInputFormat and its corresponding FixedLengthRecordReader. When creating a job that specifies this input format, the job must have the mapreduce.input.fixedlengthinputformat.record.length property set as follows myJobConf.setInt(mapreduce.input.fixedlengthinputformat.record.length,[myFixedRecordLength]); OR myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH, [myFixedRecordLength]); This input format overrides computeSplitSize() in order to ensure that InputSplits do not contain any partial records since with fixed records there is no way to determine where a record begins if that were to occur. Each InputSplit passed to the FixedLengthRecordReader will start at the beginning of a record, and the last byte in the InputSplit will be the last byte of a record. The override of computeSplitSize() delegates to FileInputFormat's compute method, and then adjusts the returned split size by doing the following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength) * fixedRecordLength) This suite of fixed length input format classes, does not support compressed files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726278#comment-13726278 ] Stanislav Barton commented on MAPREDUCE-5399: - I have the patch for the TRUNK version (the build created 3.0.0-SNAPSHOT), but I checked out the code of the two proposed versions (1.1.0 and 2.0.2-a) and the patch for trunk is applicable only to the latter. Should I create the patch for the former as well? Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream,
[jira] [Commented] (MAPREDUCE-2818) Rest API for retrieving job / task statistics
[ https://issues.apache.org/jira/browse/MAPREDUCE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726281#comment-13726281 ] Mikhail Antonov commented on MAPREDUCE-2818: Do you mean it's not going to be implemented for MRv1 job tracker? Rest API for retrieving job / task statistics -- Key: MAPREDUCE-2818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2818 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Florian Leibert Priority: Trivial Fix For: 2.0.0-alpha Attachments: HADOOP-4559v2.patch Original Estimate: 2h Remaining Estimate: 2h a rest api that returns a simple JSON containing information about a given job such as: min/max/avg times per task, failed tasks, etc. This would be useful in order to allow external restart or modification of parameters of a run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Barton updated MAPREDUCE-5399: Release Note: Fixes blank Configuration object creation overhead by reusing the Job configuration in InMemoryReader. Status: Patch Available (was: Reopened) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.0.2-alpha, 1.1.0 Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) -
[jira] [Updated] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response
[ https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leitao Guo updated MAPREDUCE-3992: -- Fix Version/s: (was: 1.1.0) Reduce fetcher doesn't verify HTTP status code of response -- Key: MAPREDUCE-3992 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.23.1, 0.24.0, 1.0.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.3, 2.0.2-alpha Attachments: mr-3992-branch-1.txt, mr-3992.txt Currently, the reduce fetch code doesn't check the HTTP status code of the response. This can lead to the following situation: - the map output servlet gets an IOException after setting the headers but before the first call to flush() - this causes it to send a response with a non-OK result code, including the exception text as the response body (response.sendError() does this if the response isn't committed) - it will still include the response headers indicating it's a valid response In the case of a merge-to-memory, the compression codec might then try to interpret the HTML response as compressed data, resulting in either a huge allocation (OOME) or some other nasty error. This bug seems to be present in MR1, but haven't checked trunk/MR2 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
Rohith Sharma K S created MAPREDUCE-5441: Summary: JobClient exit whenever RM issue Reboot command to 1st attempt App Master. Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.1-beta Reporter: Rohith Sharma K S When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726469#comment-13726469 ] Jason Lowe commented on MAPREDUCE-5399: --- I don't see the trunk patch attached to this JIRA. Could you attach it? Once it's attached, the Jenkins bot will notice it and can comment on it, and others can review it as well. Usually we iterate on the trunk patch before worrying about other branches, because normally changes go through trunk and later versions before earlier versions. This helps prevent the situation where an older Hadoop version has the fix but it's missing from a later version. Often the trunk patch will apply to the 2.x line as-is, and we can work on the 1.x patch after iterating on the trunk version. Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean)
[jira] [Updated] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Barton updated MAPREDUCE-5399: Attachment: MAPREDUCE-5399.patch Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical Attachments: MAPREDUCE-5399.patch We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) @bci=60, line=303 (Interpreted frame) - org.apache.hadoop.mapred.IFile$InMemoryReader.init(org.apache.hadoop.mapred.RamManager,
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[
https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726494#comment-13726494
]
Hadoop QA commented on MAPREDUCE-5399:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12595426/MAPREDUCE-5399.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3926//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3926//console
This message is automatically generated.
Large number of map tasks cause slow sort at reduce phase, invariant to
amount of data to sort
--
Key: MAPREDUCE-5399
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Stanislav Barton
Assignee: Stanislav Barton
Priority: Critical
Attachments: MAPREDUCE-5399.patch
We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from
4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job
in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on
input each about 100MB) and 6 000 reducers (one reducer per table region). I
was trying to figure out what at which phase the slow down appears (firstly I
suspected that the slow gathering of the 1 map output files is the
culprit) and found out that the problem is not reading the map output (the
shuffle) but the sort/merge phase that follows - the last and actual reduce
phase is fast. I have tried to up the io.sort.factor because I thought the
lots of small files are being merged on disk, but again upping that to 1000
didnt do any difference. I have then printed the stack trace and found out
that the problem is initialization of the
org.apache.hadoop.mapred.IFileInputStream namely the creation of the
Configuration object which is not propagated along from earlier context, see
the stack trace:
Thread 13332: (state = IN_NATIVE)
- java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0
(Compiled frame; information may be imprecise)
- java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228
(Compiled frame)
- java.io.File.exists() @bci=20, line=733 (Compiled frame)
- sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean)
@bci=136, line=999 (Compiled frame)
- sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean)
@bci=3, line=966 (Compiled frame)
- sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17,
line=146 (Compiled frame)
- java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame)
-
java.security.AccessController.doPrivileged(java.security.PrivilegedAction,
java.security.AccessControlContext) @bci=0 (Compiled frame)
- java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382
(Compiled frame)
- java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002
(Compiled frame)
- java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2,
line=1192 (Compiled frame)
- javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame)
-
java.security.AccessController.doPrivileged(java.security.PrivilegedAction)
@bci=0 (Compiled frame)
-
javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader,
java.lang.String) @bci=10, line=89 (Compiled frame)
-
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726500#comment-13726500 ] Stanislav Barton commented on MAPREDUCE-5399: - In the proposed patch, I have replaced the constructor that allowed passing no Configuration object, then looked for all usages of the removed constructor and fixed the call by adding the Configuration object already known from the context. To my opinion, if the code compiles and the tests pass, it should be good to go, since the new constructor is not backwards compatible. Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical Attachments: MAPREDUCE-5399.patch We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration)
[jira] [Commented] (MAPREDUCE-2818) Rest API for retrieving job / task statistics
[ https://issues.apache.org/jira/browse/MAPREDUCE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726519#comment-13726519 ] Harsh J commented on MAPREDUCE-2818: Mikhail, See MAPREDUCE-4837 perhaps. Rest API for retrieving job / task statistics -- Key: MAPREDUCE-2818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2818 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Florian Leibert Priority: Trivial Fix For: 2.0.0-alpha Attachments: HADOOP-4559v2.patch Original Estimate: 2h Remaining Estimate: 2h a rest api that returns a simple JSON containing information about a given job such as: min/max/avg times per task, failed tasks, etc. This would be useful in order to allow external restart or modification of parameters of a run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5352) Optimize node local splits generated by CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726646#comment-13726646 ] Siddharth Seth commented on MAPREDUCE-5352: --- Thanks for taking a look Bikas and Devaraj. Committing this. Optimize node local splits generated by CombineFileInputFormat --- Key: MAPREDUCE-5352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5352 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5352.1.txt, MAPREDUCE-5352.2.txt, MAPREDUCE-5352.3.txt, MAPREDUCE-5352.4.txt CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse. Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5352) Optimize node local splits generated by CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5352: -- Attachment: MAPREDUCE-5352.4.branch-2.1-beta.txt Patch for branch 2.1-beta with a minor conflict resolved. Optimize node local splits generated by CombineFileInputFormat --- Key: MAPREDUCE-5352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5352 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MAPREDUCE-5352.1.txt, MAPREDUCE-5352.2.txt, MAPREDUCE-5352.3.txt, MAPREDUCE-5352.4.branch-2.1-beta.txt, MAPREDUCE-5352.4.txt CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse. Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5352) Optimize node local splits generated by CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5352: -- Resolution: Fixed Fix Version/s: 2.1.1-beta Target Version/s: (was: 2.1.0-beta) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-2.1-beta. Optimize node local splits generated by CombineFileInputFormat --- Key: MAPREDUCE-5352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5352 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.1.1-beta Attachments: MAPREDUCE-5352.1.txt, MAPREDUCE-5352.2.txt, MAPREDUCE-5352.3.txt, MAPREDUCE-5352.4.branch-2.1-beta.txt, MAPREDUCE-5352.4.txt CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse. Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5352) Optimize node local splits generated by CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726672#comment-13726672 ] Hudson commented on MAPREDUCE-5352: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4198 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4198/]) MAPREDUCE-5352. Optimize node local splits generated by CombineFileInputFormat. (sseth) (sseth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1509345) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java Optimize node local splits generated by CombineFileInputFormat --- Key: MAPREDUCE-5352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5352 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.5-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.1.1-beta Attachments: MAPREDUCE-5352.1.txt, MAPREDUCE-5352.2.txt, MAPREDUCE-5352.3.txt, MAPREDUCE-5352.4.branch-2.1-beta.txt, MAPREDUCE-5352.4.txt CombineFileInputFormat currently walks through all available nodes and generates multiple (maxSplitsPerNode) splits on a single node before attempting to generate splits on subsequent nodes. This ends up reducing the possibility of generating splits for subsequent nodes - since these blocks will no longer be available for subsequent nodes. Allowing splits to go 1 block above the max-split-size makes this worse. Allocating a single split per node in one iteration, should help increase the distribution of splits across nodes - so the subsequent nodes will have more blocks to choose from. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726796#comment-13726796 ] Jason Lowe commented on MAPREDUCE-5428: --- +1, lgtm. Committing shortly. HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5379: -- Attachment: MAPREDUCE-5379-2.patch Include FS delegation token ID in job conf -- Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission, security Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, MAPREDUCE-5379.patch Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726806#comment-13726806 ] Hudson commented on MAPREDUCE-5428: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4200 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4200/]) MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is stopped. Contributed by Karthik Kambatla (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1509401) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[
https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726815#comment-13726815
]
Hadoop QA commented on MAPREDUCE-5379:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12595472/MAPREDUCE-5379-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3927//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3927//console
This message is automatically generated.
Include FS delegation token ID in job conf
--
Key: MAPREDUCE-5379
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: job submission, security
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch,
MAPREDUCE-5379.patch
Making a job's FS delegation token ID accessible will allow external services
to associate it with the file system operations it performs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5379: -- Status: Patch Available (was: Open) Include FS delegation token ID in job conf -- Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission, security Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, MAPREDUCE-5379.patch Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5428: -- Resolution: Fixed Fix Version/s: 2.1.0-beta 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Karthik! I committed this to trunk, branch-2, and branch-2.1-beta. HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Fix For: 3.0.0, 2.1.0-beta Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726830#comment-13726830 ] Alejandro Abdelnur commented on MAPREDUCE-5379: --- I've played around and got Daryn's patch to work. After running the patch by Andrew Wang (who is doing HDFS-4680) he brought up a a concern with the client driven tracking approach, a client can set a rogue trackingId. But with the sequenceId approach what is in the HDFS audit can fully trusted and tracked to a user. One concern Daryn mentioned above with the sequenceId approach was, and also told me offline, the MR client decoding the token identifier, this could break things when moving token encoding from writable to protobuff. To address this, instead of the MR client decoding the token identifier it would simply do a hash of its byte[] representation without decoding it. In addition, the MR client should have an option to switch ON/OFF(default) the DT hash generation/injection in the jobconf. Include FS delegation token ID in job conf -- Key: MAPREDUCE-5379 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission, security Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, MAPREDUCE-5379.patch Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4830) Restore JT DelegationTokenSecretManager state on restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned MAPREDUCE-4830: --- Assignee: Karthik Kambatla Restore JT DelegationTokenSecretManager state on restart Key: MAPREDUCE-4830 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4830 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.0 Reporter: Tom White Assignee: Karthik Kambatla This is the MR1 equivalent of YARN-248. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5440) TestCopyCommitter Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5440: - Attachment: MAPREDUCE-5440.patch TestCopyCommitter Fails on JDK7 --- Key: MAPREDUCE-5440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5440 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk, 2.3.0 Reporter: Robert Parker Assignee: Robert Parker Attachments: MAPREDUCE-5440.patch The testNoCommitAction is affected by the testPreserveStatus. The testPreserveStatus leaves the CONF_LABEL_PRESERVE_STATUS set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5440) TestCopyCommitter Fails on JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5440: - Status: Patch Available (was: Open) TestCopyCommitter Fails on JDK7 --- Key: MAPREDUCE-5440 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5440 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk, 2.3.0 Reporter: Robert Parker Assignee: Robert Parker Attachments: MAPREDUCE-5440.patch The testNoCommitAction is affected by the testPreserveStatus. The testPreserveStatus leaves the CONF_LABEL_PRESERVE_STATUS set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5440) TestCopyCommitter Fails on JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726917#comment-13726917
]
Hadoop QA commented on MAPREDUCE-5440:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12595483/MAPREDUCE-5440.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3928//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3928//console
This message is automatically generated.
TestCopyCommitter Fails on JDK7
---
Key: MAPREDUCE-5440
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5440
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: trunk, 2.3.0
Reporter: Robert Parker
Assignee: Robert Parker
Attachments: MAPREDUCE-5440.patch
The testNoCommitAction is affected by the testPreserveStatus. The
testPreserveStatus leaves the CONF_LABEL_PRESERVE_STATUS set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726954#comment-13726954 ] Jian He commented on MAPREDUCE-5441: Debugged a while. if you are running on MAC, you probably will get this problem (see YARN-76). If you are running on Linux, you should not get problem. The reason is: After RM restarts, before Reboot command is actually processed by AM, AM will get AMRMToken invalid Exception, since now AMRMToken is used in non-secure mode. What MR AM now handles this exception is just ignoring it and keeping retry, essentially an infinite loop. On linux, AM process will be quickly killed by NM sending the signal, RM launches a new AM, during this time JobClient will retry and eventually switched to the new AM; But on MAC, AM process is probably still hanging around. This leads to the JobClient keeps talking with the old AM, the old AM will eventually tell the Client that the job failed. Tested this on real cluster and see that JobClient will hang a while and eventually continues reporting job progress. JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.1-beta Reporter: Rohith Sharma K S When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort
[ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727046#comment-13727046 ] Sandy Ryza commented on MAPREDUCE-5399: --- The patch looks good to me. It seems like this would be difficult to write a test for. Have you done any benchmarking to see if/how much it improves performance? Large number of map tasks cause slow sort at reduce phase, invariant to amount of data to sort -- Key: MAPREDUCE-5399 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 1.1.0, 2.0.2-alpha Reporter: Stanislav Barton Assignee: Stanislav Barton Priority: Critical Attachments: MAPREDUCE-5399.patch We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our MR job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 files on input each about 100MB) and 6 000 reducers (one reducer per table region). I was trying to figure out what at which phase the slow down appears (firstly I suspected that the slow gathering of the 1 map output files is the culprit) and found out that the problem is not reading the map output (the shuffle) but the sort/merge phase that follows - the last and actual reduce phase is fast. I have tried to up the io.sort.factor because I thought the lots of small files are being merged on disk, but again upping that to 1000 didnt do any difference. I have then printed the stack trace and found out that the problem is initialization of the org.apache.hadoop.mapred.IFileInputStream namely the creation of the Configuration object which is not propagated along from earlier context, see the stack trace: Thread 13332: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean) @bci=136, line=999 (Compiled frame) - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolean) @bci=3, line=966 (Compiled frame) - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=17, line=146 (Compiled frame) - java.net.URLClassLoader$2.run() @bci=12, line=385 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame) - java.net.URLClassLoader.findResource(java.lang.String) @bci=13, line=382 (Compiled frame) - java.lang.ClassLoader.getResource(java.lang.String) @bci=30, line=1002 (Compiled frame) - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=2, line=1192 (Compiled frame) - javax.xml.parsers.SecuritySupport$4.run() @bci=26, line=96 (Compiled frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame) - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassLoader, java.lang.String) @bci=10, line=89 (Compiled frame) - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.String) @bci=38, line=250 (Interpreted frame) - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) @bci=273, line=223 (Interpreted frame) - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=4, line=123 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=16, line=1890 (Compiled frame) - org.apache.hadoop.conf.Configuration.loadResources(java.util.Properties, java.util.ArrayList, boolean) @bci=49, line=1867 (Compiled frame) - org.apache.hadoop.conf.Configuration.getProps() @bci=43, line=1785 (Compiled frame) - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=35, line=712 (Compiled frame) - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=2, line=731 (Compiled frame) - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, boolean) @bci=2, line=1047 (Interpreted frame) - org.apache.hadoop.mapred.IFileInputStream.init(java.io.InputStream, long, org.apache.hadoop.conf.Configuration) @bci=111, line=93 (Interpreted frame) - org.apache.hadoop.mapred.IFile$Reader.init(org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoop.io.compress.CompressionCodec,
[jira] [Created] (MAPREDUCE-5442) $HADOOP_MAPRED_HOME setting not working on Windows
Yingda Chen created MAPREDUCE-5442: -- Summary: $HADOOP_MAPRED_HOME setting not working on Windows Key: MAPREDUCE-5442 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5442 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yingda Chen Assignee: Yingda Chen Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5442) $HADOOP_MAPRED_HOME setting not working on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727057#comment-13727057 ] Yingda Chen commented on MAPREDUCE-5442: This is another environment issue on Windows, which is quite similar to YARN-729. Currently the mapred-default.xml has mapreduce.application.classpath entry set to value$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value which is problematic on Windows since the path does not work on Windows OS, instead we should be using %HADOOP_MAPRED_HOME%\\share\\hadoop\\mapreduce\\ for Windows. $HADOOP_MAPRED_HOME setting not working on Windows -- Key: MAPREDUCE-5442 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5442 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Yingda Chen Assignee: Yingda Chen Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5428: -- Fix Version/s: (was: 2.1.0-beta) 2.1.1-beta HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Fix For: 3.0.0, 2.1.1-beta Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5251: -- Fix Version/s: (was: 2.3.0) 2.1.1-beta I pulled this into branch-2.1-beta as well. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Fix For: 3.0.0, 0.23.10, 2.1.1-beta Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt, MAPREDUCE-5251-7-b23.txt, MAPREDUCE-5251-7.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5428) HistoryFileManager doesn't stop threads when service is stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727137#comment-13727137 ] Hudson commented on MAPREDUCE-5428: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4203 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4203/]) Correct fix versions for MAPREDUCE-5428 and YARN-573. (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1509475) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt HistoryFileManager doesn't stop threads when service is stopped --- Key: MAPREDUCE-5428 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5428 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.0.4-alpha Reporter: Jason Lowe Assignee: Karthik Kambatla Fix For: 3.0.0, 2.1.1-beta Attachments: mr-5428-1.patch HistoryFileManager is a service that starts threads, but it does not override the serviceStop method to stop the threads when the service is stopped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5317:
--
Fix Version/s: (was: 2.3.0)
2.1.1-beta
I pulled this into branch-2.1-beta as well.
Stale files left behind for failed jobs
---
Key: MAPREDUCE-5317
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Fix For: 3.0.0, 0.23.10, 2.1.1-beta
Attachments: MAPREDUCE-5317.branch-0.23.patch,
MAPREDUCE-5317.branch-0.23.patch, MAPREDUCE-5317.branch-0.23.patch,
MAPREDUCE-5317.patch, MAPREDUCE-5317.patch, MAPREDUCE-5317.patch,
MAPREDUCE-5317.patch, MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
Courtesy [~amar_kamat]!
{quote}
We are seeing _temporary files left behind in the output folder if the job
fails.
The job were failed due to hitting quota issue.
I simply ran the randomwriter (from hadoop examples) with the default setting.
That failed and left behind some stray files.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5358) MRAppMaster throws invalid transitions for JobImpl
[
https://issues.apache.org/jira/browse/MAPREDUCE-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5358:
--
Fix Version/s: (was: 2.3.0)
2.1.1-beta
I pulled this into branch-2.1-beta as well.
MRAppMaster throws invalid transitions for JobImpl
--
Key: MAPREDUCE-5358
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5358
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.0.1-alpha, 2.0.5-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 3.0.0, 2.1.1-beta
Attachments: MAPREDUCE-5358.patch
{code:xml}
2013-06-26 11:39:50,128 ERROR [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_TASK_ATTEMPT_COMPLETED at SUCCEEDED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:720)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:962)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:958)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-06-26 11:39:50,129 ERROR [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event
at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_MAP_TASK_RESCHEDULED at SUCCEEDED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:720)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:962)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:958)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read files recursively in the input path dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3193: -- Fix Version/s: (was: 2.3.0) 2.1.1-beta I pulled this into branch-2.1-beta as well. FileInputFormat doesn't read files recursively in the input path dir Key: MAPREDUCE-3193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.2, 2.0.0-alpha, 3.0.0 Reporter: Ramgopal N Assignee: Devaraj K Fix For: 3.0.0, 0.23.10, 2.1.1-beta Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193-4.patch, MAPREDUCE-3193-5.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed. Example:Input file is /r1/r2/input.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5443) ClientId should have getMsb/getLsb methods
Tsuyoshi OZAWA created MAPREDUCE-5443: - Summary: ClientId should have getMsb/getLsb methods Key: MAPREDUCE-5443 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5443 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Both ClientId and RetryCache have the same logic to calculate msb and lsb. We should not have same logics in separate classes but have utility methods to do so in one class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5443) ClientId should have getMsb/getLsb methods
[ https://issues.apache.org/jira/browse/MAPREDUCE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA resolved MAPREDUCE-5443. --- Resolution: Invalid Release Note: Moved to HADOOP-9821, and close this as invalid, because this is a problem about hadoop-common. ClientId should have getMsb/getLsb methods -- Key: MAPREDUCE-5443 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5443 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Both ClientId and RetryCache have the same logic to calculate msb and lsb. We should not have same logics in separate classes but have utility methods to do so in one class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13727367#comment-13727367 ] Rohith Sharma K S commented on MAPREDUCE-5441: -- Jian, thank you for your brief explanation. I am getting this problem in Linux machine. The scenario is app master running Node Manager is abruptly killed(kill -9 NM_PID) and restarted. In the above test case, RM issues reboot to 1st attempt app master. At this time, job status is set as ERROR and trigger JobHistoryEvent. Here, still Jobclient is connecting to 1st app master for getting job report and client get job report with jobstatus as ERROR. JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.1-beta Reporter: Rohith Sharma K S When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5441) JobClient exit whenever RM issue Reboot command to 1st attempt App Master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated MAPREDUCE-5441: - Affects Version/s: 2.1.0-beta 2.0.5-alpha JobClient exit whenever RM issue Reboot command to 1st attempt App Master. -- Key: MAPREDUCE-5441 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5441 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, client Affects Versions: 2.1.0-beta, 2.0.5-alpha, 2.1.1-beta Reporter: Rohith Sharma K S When RM issue Reboot command to app master, app master shutdown gracefully. All the history event are writtent to hdfs with job status set as ERROR. Jobclient get job state as ERROR and exit. But RM launches 2nd attempt app master where no client are there to get job status.In RM UI, job status is displayed as SUCCESS but for client Job is Failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
