[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13903999#comment-13903999
]
Liyin Liang commented on MAPREDUCE-5487:
The following line is not necessary any more. It should be deleted.
{code}
static final Configuration conf = new JobConf();
{code}
In task processes, JobConf is unnecessarily loaded again in Limits
--
Key: MAPREDUCE-5487
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Fix For: 2.3.0
Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch
Limits statically loads a JobConf, which incurs costs of reading files from
disk and parsing XML. The contents of this JobConf are identical to the one
loaded by YarnChild (before adding job.xml as a resource). Allowing Limits
to initialize with the JobConf loaded in YarnChild would reduce task startup
time.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904300#comment-13904300 ] Sandy Ryza commented on MAPREDUCE-5487: --- Filed MAPREDUCE-5759 In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904299#comment-13904299 ] Sandy Ryza commented on MAPREDUCE-5487: --- Very good point. Not sure how I missed that. In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838320#comment-13838320 ] Hitesh Shah commented on MAPREDUCE-5487: A little bit late on this. Did anyone look into how this affects jobs where a user modifies the counter limit to be higher than the cluster configured value and what happens in the case where the jobhistory server is configured with a limit less than the user supplied limit? In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838478#comment-13838478 ] Sandy Ryza commented on MAPREDUCE-5487: --- Yup, there should be no changes in those cases I missed something. In task processes, we create a JobConf with the cluster configs, use it to initialize Limits, and then load the user configs. So the user configs do not have a chance to affect Limits. In all other cases, the limits are loaded from the cluster config exactly as before. In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838525#comment-13838525 ] Sandy Ryza commented on MAPREDUCE-5487: --- *unless I missed something In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.4.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771791#comment-13771791 ] Hudson commented on MAPREDUCE-5487: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #337 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/337/]) MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again in Limits (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524408) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestCounters.java In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771868#comment-13771868 ] Hudson commented on MAPREDUCE-5487: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1553 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1553/]) MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again in Limits (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524408) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestCounters.java In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771878#comment-13771878 ] Hudson commented on MAPREDUCE-5487: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1527 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1527/]) MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again in Limits (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524408) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestCounters.java In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770782#comment-13770782 ] Hudson commented on MAPREDUCE-5487: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4434 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4434/]) MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again in Limits (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1524408) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/Limits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestCounters.java In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770785#comment-13770785 ] Sandy Ryza commented on MAPREDUCE-5487: --- I just committed this In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769854#comment-13769854 ] Alejandro Abdelnur commented on MAPREDUCE-5487: --- On my first comment, my bad, mistakenly thought JOB_CONF_FILE was mapred-site.xml, it is job.xml, the localized job. It is fine then. +1 In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768302#comment-13768302
]
Alejandro Abdelnur commented on MAPREDUCE-5487:
---
Shouldn't {{Limits.init(job)}} be called after adding the mapred config as
resource?
Personally, I don't like constants that are not 'constants', that seems to be
the case of these limits. I know this is not being introduced by this patch. I
would change all code to use the methods and deprecate the constants. I'm OK
with doing that in another patch though.
In task processes, JobConf is unnecessarily loaded again in Limits
--
Key: MAPREDUCE-5487
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch
Limits statically loads a JobConf, which incurs costs of reading files from
disk and parsing XML. The contents of this JobConf are identical to the one
loaded by YarnChild (before adding job.xml as a resource). Allowing Limits
to initialize with the JobConf loaded in YarnChild would reduce task startup
time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768340#comment-13768340 ] Sandy Ryza commented on MAPREDUCE-5487: --- bq. Shouldn't Limits.init(job) be called after adding the mapred config as resource? This would diverge from the previous behavior, which uses a vanilla JobConf for loading the Limits constants. bq. Personally, I don't like constants that are not 'constants', that seems to be the case of these limits. Totally agree, though also think it should be a separate patch. In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753970#comment-13753970
]
Hadoop QA commented on MAPREDUCE-5487:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600640/MAPREDUCE-5487.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3971//console
This message is automatically generated.
In task processes, JobConf is unnecessarily loaded again in Limits
--
Key: MAPREDUCE-5487
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5487.patch
Limits statically loads a JobConf, which incurs costs of reading files from
disk and parsing XML. The contents of this JobConf are identical to the one
loaded by YarnChild (before adding job.xml as a resource). Allowing Limits
to initialize with the JobConf loaded in YarnChild would reduce task startup
time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754297#comment-13754297
]
Hadoop QA commented on MAPREDUCE-5487:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600694/MAPREDUCE-5487-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3973//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3973//console
This message is automatically generated.
In task processes, JobConf is unnecessarily loaded again in Limits
--
Key: MAPREDUCE-5487
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch
Limits statically loads a JobConf, which incurs costs of reading files from
disk and parsing XML. The contents of this JobConf are identical to the one
loaded by YarnChild (before adding job.xml as a resource). Allowing Limits
to initialize with the JobConf loaded in YarnChild would reduce task startup
time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754383#comment-13754383 ] Sandy Ryza commented on MAPREDUCE-5487: --- test failure is unrelated In task processes, JobConf is unnecessarily loaded again in Limits -- Key: MAPREDUCE-5487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch Limits statically loads a JobConf, which incurs costs of reading files from disk and parsing XML. The contents of this JobConf are identical to the one loaded by YarnChild (before adding job.xml as a resource). Allowing Limits to initialize with the JobConf loaded in YarnChild would reduce task startup time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
