[jira] [Updated] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated MAPREDUCE-6468:
Attachment: MAPREDUCE-6468-03.patch
Attaching the patch.Addressed white spaces and check style.Findbugs need to
check as no issue displayed. Test failures are not part of patch updation.
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch, HADOOP-9995.patch,
> MAPREDUCE-6468-01.patch, MAPREDUCE-6468-02.patch, MAPREDUCE-6468-03.patch
>
>
> Developers use logs to do in-house debugging. These log statements are later
> demoted to less severe levels and usually are guarded by their matching
> severity levels. However, we do see inconsistencies in trunk. A log statement
> like
> {code}
>if (LOG.isDebugEnabled()) {
> LOG.info("Assigned container (" + allocated + ") "
> {code}
> doesn't make much sense because the log message is actually only printed out
> in DEBUG-level. We do see previous issues tried to correct this
> inconsistency. I am proposing a comprehensive correction over trunk.
> Doug Cutting pointed it out in HADOOP-312:
> https://issues.apache.org/jira/browse/HADOOP-312?focusedCommentId=12429498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12429498
> HDFS-1611 also corrected this inconsistency.
> This could have been avoided by switching from log4j to slf4j's {} format
> like CASSANDRA-625 (2010/3) and ZOOKEEPER-850 (2012/1), which gives cleaner
> code and slightly higher performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736364#comment-14736364
]
Tsuyoshi Ozawa commented on MAPREDUCE-6468:
---
[~jagadesh.kiran] Could you double check that the patch you uploaded is
correct? It seems old one.
{code}
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java
@@ -374,9 +374,7 @@ private boolean checkLogsAvailableForRead(FSImage image,
long imageTxId,
"or call saveNamespace on the active node.\n" +
"Error: " + e.getLocalizedMessage();
if (LOG.isDebugEnabled()) {
-LOG.fatal(msg, e);
- } else {
-LOG.fatal(msg);
+LOG.debug("", e);
}
return false;
}
{code}
Also, in BackupImage, the log level looks wrong. Please fix it.
{code}
if (LOG.isDebugEnabled()) {
- LOG.debug("State transition " + bnState + " -> " + newState);
+ LOG.trace("State transition " + bnState + " -> " + newState);
}
{code}
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch, HADOOP-9995.patch,
> MAPREDUCE-6468-01.patch, MAPREDUCE-6468-02.patch, MAPREDUCE-6468-03.patch
>
>
> Developers use logs to do in-house debugging. These log statements are later
> demoted to less severe levels and usually are guarded by their matching
> severity levels. However, we do see inconsistencies in trunk. A log statement
> like
> {code}
>if (LOG.isDebugEnabled()) {
> LOG.info("Assigned container (" + allocated + ") "
> {code}
> doesn't make much sense because the log message is actually only printed out
> in DEBUG-level. We do see previous issues tried to correct this
> inconsistency. I am proposing a comprehensive correction over trunk.
> Doug Cutting pointed it out in HADOOP-312:
> https://issues.apache.org/jira/browse/HADOOP-312?focusedCommentId=12429498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12429498
> HDFS-1611 also corrected this inconsistency.
> This could have been avoided by switching from log4j to slf4j's {} format
> like CASSANDRA-625 (2010/3) and ZOOKEEPER-850 (2012/1), which gives cleaner
> code and slightly higher performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736315#comment-14736315
]
Tsuyoshi Ozawa commented on MAPREDUCE-6468:
---
[~jagadesh.kiran] thank you for updating. In BootstrapStandby, the message
should be used instead of passing Exception object directly since it helps
users to understand what's happened.
{code}
@@ -374,9 +374,7 @@ private boolean checkLogsAvailableForRead(FSImage image,
long imageTxId,
"or call saveNamespace on the active node.\n" +
"Error: " + e.getLocalizedMessage();
if (LOG.isDebugEnabled()) {
-LOG.fatal(msg, e);
- } else {
-LOG.fatal(msg);
+LOG.debug("", e);
}
{code}
Also, the log looks fatal one: we shouldn't remove it. Could you update it as
follows?:
{code}
if (LOG.isDebugEnabled()) {
LOG.log(msg, e); // this line should be fixed as LOG.debug
} else {
LOG.fatal(msg); // this line and else statement should remain here.
}
{code}
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch, HADOOP-9995.patch,
> MAPREDUCE-6468-01.patch, MAPREDUCE-6468-02.patch
>
>
> Developers use logs to do in-house debugging. These log statements are later
> demoted to less severe levels and usually are guarded by their matching
> severity levels. However, we do see inconsistencies in trunk. A log statement
> like
> {code}
>if (LOG.isDebugEnabled()) {
> LOG.info("Assigned container (" + allocated + ") "
> {code}
> doesn't make much sense because the log message is actually only printed out
> in DEBUG-level. We do see previous issues tried to correct this
> inconsistency. I am proposing a comprehensive correction over trunk.
> Doug Cutting pointed it out in HADOOP-312:
> https://issues.apache.org/jira/browse/HADOOP-312?focusedCommentId=12429498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12429498
> HDFS-1611 also corrected this inconsistency.
> This could have been avoided by switching from log4j to slf4j's {} format
> like CASSANDRA-625 (2010/3) and ZOOKEEPER-850 (2012/1), which gives cleaner
> code and slightly higher performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated MAPREDUCE-6468:
Attachment: MAPREDUCE-6468-04.patch
[~ozawa] thanks for your review, addressed all your comments in this patch
,please review
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch, HADOOP-9995.patch,
> MAPREDUCE-6468-01.patch, MAPREDUCE-6468-02.patch, MAPREDUCE-6468-03.patch,
> MAPREDUCE-6468-04.patch
>
>
> Developers use logs to do in-house debugging. These log statements are later
> demoted to less severe levels and usually are guarded by their matching
> severity levels. However, we do see inconsistencies in trunk. A log statement
> like
> {code}
>if (LOG.isDebugEnabled()) {
> LOG.info("Assigned container (" + allocated + ") "
> {code}
> doesn't make much sense because the log message is actually only printed out
> in DEBUG-level. We do see previous issues tried to correct this
> inconsistency. I am proposing a comprehensive correction over trunk.
> Doug Cutting pointed it out in HADOOP-312:
> https://issues.apache.org/jira/browse/HADOOP-312?focusedCommentId=12429498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12429498
> HDFS-1611 also corrected this inconsistency.
> This could have been avoided by switching from log4j to slf4j's {} format
> like CASSANDRA-625 (2010/3) and ZOOKEEPER-850 (2012/1), which gives cleaner
> code and slightly higher performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736761#comment-14736761
]
Hadoop QA commented on MAPREDUCE-6468:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 25m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 9m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 15s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 6m 37s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 10m 39s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 41s | Pre-build of native portion |
| {color:green}+1{color} | mapreduce tests | 9m 50s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | tools/hadoop tests | 6m 59s | Tests passed in
hadoop-distcp. |
| {color:red}-1{color} | tools/hadoop tests | 10m 14s | Tests failed in
hadoop-gridmix. |
| {color:red}-1{color} | yarn tests | 5m 32s | Tests failed in
hadoop-yarn-client. |
| {color:green}+1{color} | yarn tests | 1m 59s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 54m 34s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| {color:red}-1{color} | hdfs tests | 162m 40s | Tests failed in hadoop-hdfs. |
| | | 323m 0s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA |
| | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Timed out tests | org.apache.hadoop.mapred.gridmix.TestGridmixSubmission |
| | org.apache.hadoop.mapred.gridmix.TestSleepJob |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754844/MAPREDUCE-6468-03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 94cf7ab |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-distcp test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-distcp.txt
|
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5975/console |
This message was automatically generated.
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>
[jira] [Commented] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736793#comment-14736793
]
Jagadesh Kiran N commented on MAPREDUCE-6468:
-
No Findbugs in the Report, And the test case failures are not related to the
Patch.
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch, HADOOP-9995.patch,
> MAPREDUCE-6468-01.patch, MAPREDUCE-6468-02.patch, MAPREDUCE-6468-03.patch,
> MAPREDUCE-6468-04.patch
>
>
> Developers use logs to do in-house debugging. These log statements are later
> demoted to less severe levels and usually are guarded by their matching
> severity levels. However, we do see inconsistencies in trunk. A log statement
> like
> {code}
>if (LOG.isDebugEnabled()) {
> LOG.info("Assigned container (" + allocated + ") "
> {code}
> doesn't make much sense because the log message is actually only printed out
> in DEBUG-level. We do see previous issues tried to correct this
> inconsistency. I am proposing a comprehensive correction over trunk.
> Doug Cutting pointed it out in HADOOP-312:
> https://issues.apache.org/jira/browse/HADOOP-312?focusedCommentId=12429498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12429498
> HDFS-1611 also corrected this inconsistency.
> This could have been avoided by switching from log4j to slf4j's {} format
> like CASSANDRA-625 (2010/3) and ZOOKEEPER-850 (2012/1), which gives cleaner
> code and slightly higher performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736983#comment-14736983 ] Jason Lowe commented on MAPREDUCE-6415: --- +1 latest patch looks good to me. Will commit this later today if there are no objections. > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736993#comment-14736993
]
Sunil G commented on MAPREDUCE-5870:
Thank you [~jlowe] for sharing the comments.
As discussed offline, I had doubts on couple of points here.
bq.We are already returning the job priority via getJobStatus
As I see, we are invoking {{YARNRunner#getJobStatus}} to get the job status
while we try to call {{Job#getPriority}}.
And in {{ClientServiceDelegate#getJobStatus}}, *JobStatus* is created from
*JobReport* (from AM).
{code}
public JobStatus getJobStatus(JobID oldJobID) throws IOException {
org.apache.hadoop.mapreduce.v2.api.records.JobId jobId =
TypeConverter.toYarn(oldJobID);
GetJobReportRequest request =
recordFactory.newRecordInstance(GetJobReportRequest.class);
request.setJobId(jobId);
JobReport report = ((GetJobReportResponse) invoke("getJobReport",
GetJobReportRequest.class, request)).getJobReport();
...
...
jobStatus = TypeConverter.fromYarn(report, url);
...
{code}
As this JobReport doesnt have priority information, current code in
{{TypeConverter#fromYarn}} is to hardcode the status.
{code}
public static JobStatus fromYarn(JobReport jobreport, String trackingUrl) {
JobPriority jobPriority = JobPriority.NORMAL;
JobStatus jobStatus = new org.apache.hadoop.mapred.JobStatus(
fromYarn(jobreport.getJobId()), jobreport.getSetupProgress(), jobreport
{code}
AppReport has the information about the priority of the application. So a
second call has to be made to fetch the same from {{YarnRunner#getJobStatus}}.
And it can be set to JobStatus separately.
Kindly help to share your thoughts here if I am missing something important.
bq.Why was the call to testChangingJobPriority removed from TestMRJobClient?
{{TestMRJobClient#testChangingJobPriority}} was invoked after job was
completed. And YARN throws excpetion for change priority application if its not
RUNNING or ACCEPTED. As updated offline, I agree that YARN need not have to
throw excpetion if the app is completed, rather it can be skipped. I will file
a ticket in YARN to fix the same. Thoughts?
For LocalJobRunner exception case, I will file another ticket to fix the
exception and to support correct priority.
Also I will add all possible test cases for Job and JobConf to see all possible
integer and enums are set/get correctly.
Thank you for the support.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737035#comment-14737035 ] Jason Lowe commented on MAPREDUCE-5870: --- bq. AppReport has the information about the priority of the application. So a second call has to be made to fetch the same from YarnRunner#getJobStatus. And it can be set to JobStatus separately. The problem is that the ApplicationMaster doesn't actually know what its job priority actually is, as that's a concept tracked by the RM not the AM. So to report the priority we'd have to make a separate call to the RM to determine the priority and meld that into the rest of the information returned for the job status. I'm not thrilled with the thought of adding an additional call to the RM for every time a client wants to get the job status, especially for a field that most clients won't need. Clients often bang on the job status to see when a job is done, and that means they'll start banging on the RM as well if we make the delegate start hitting both the AM and the RM to generate the JobStatus. There's only two ways I can currently think of to get around that. Either we need to obsolete the priority field from the job status and provide a separate call to get the priority when the client really wants to know it, or we have to find a way for the AM to know its job priority so it can return it in its JobStatus responses. For the latter we could have the RM send it down in heartbeat responses, but there would be a delay between when a client updates the priority and the AM reports the updated value. We could have the AM proactively ask the RM what its priority is, but then we're back to banging on the RM for every job status call (only this time from AMs instead of the client, but same number of RM calls overall). Any other thoughts on what we should do here? I'm not thrilled about the thought of ditching the priority in the JobStatus field. However it's been worthless ever since we went to YARN (i.e.: broken since 0.23 and 2.x), so it's tempting to declare it officially obsolete (as it in practice already is) and provide a different API to retrieve it. bq. And YARN throws excpetion for change priority application if its not RUNNING or ACCEPTED. Yes, this to me is a bug. We don't throw exceptions when clients try to kill apps that have terminated, and similarly we shouldn't throw exceptions if clients try to change the priority of terminated apps. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737076#comment-14737076
]
Eric Payne commented on MAPREDUCE-5870:
---
{quote}
There's only two ways I can currently think of to get around that. Either we
need to obsolete the priority field from the job status and provide a separate
call to get the priority when the client really wants to know it, or we have to
find a way for the AM to know its job priority so it can return it in its
JobStatus responses. For the latter we could have the RM send it down in
heartbeat responses, but there would be a delay between when a client updates
the priority and the AM reports the updated value.
{quote}
[~jlowe] and [~sunilg],
I would vote for the second option. That is, update the priority in the RM and
then tell the AM about it when it hearbeats in. I think it would be fine to
have the short delay between when the priority was updated and the AM knows
about it. I can't foresee a use case where the client would atomically need to
know about the change. The one thing that might cause confusion is if the
client set the priority and immediately read the priority which might still be
the old value. This may cause confusion, but as long as this behavior is well
documented it should be fine.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6468) Consistent log severity level guards and statements in MapReduce project
[
https://issues.apache.org/jira/browse/MAPREDUCE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737006#comment-14737006
]
Hadoop QA commented on MAPREDUCE-6468:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 22m 59s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 1s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 12s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 4m 55s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 9m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | mapreduce tests | 8m 52s | Tests failed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | tools/hadoop tests | 6m 23s | Tests passed in
hadoop-distcp. |
| {color:green}+1{color} | tools/hadoop tests | 14m 48s | Tests passed in
hadoop-gridmix. |
| {color:green}+1{color} | yarn tests | 6m 56s | Tests passed in
hadoop-yarn-client. |
| {color:red}-1{color} | yarn tests | 0m 20s | Tests failed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 0m 20s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| {color:red}-1{color} | hdfs tests | 162m 1s | Tests failed in hadoop-hdfs. |
| | | 260m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider |
| | hadoop.mapreduce.v2.app.TestJobEndNotifier |
| | hadoop.mapreduce.v2.app.TestMRApp |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| Timed out tests | org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
|
| Failed build | hadoop-yarn-common |
| | hadoop-yarn-server-resourcemanager |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754880/MAPREDUCE-6468-04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0113e45 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-distcp test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-distcp.txt
|
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| hadoop-yarn-client test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-yarn-client.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5976/console |
This message was automatically generated.
> Consistent log severity level guards and statements in MapReduce project
>
>
> Key: MAPREDUCE-6468
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6468
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Jackie Chang
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9995-00.patch,
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737892#comment-14737892 ] Karthik Kambatla commented on MAPREDUCE-6415: - Checking this in. (Jason, sorry for the jumping the gun here). > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738006#comment-14738006 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-Yarn-trunk #1102 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1102/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * MAPREDUCE-6415.003.patch * hadoop-mapreduce-project/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-mapreduce-project/bin/mapred * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/pom.xml * hadoop-project/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-tools/hadoop-tools-dist/pom.xml > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737909#comment-14737909 ] Robert Kanter commented on MAPREDUCE-6415: -- Thanks everyone! I'm glad we finally have a workable solution to this issue in now > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737978#comment-14737978 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-trunk-Commit #8426 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8426/]) removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737907#comment-14737907 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-trunk-Commit #8424 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8424/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-mapreduce-project/CHANGES.txt * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-mapreduce-project/bin/mapred * MAPREDUCE-6415.003.patch * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-project/pom.xml * hadoop-tools/hadoop-archive-logs/pom.xml > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737906#comment-14737906 ] Karthik Kambatla commented on MAPREDUCE-6415: - Committed to trunk and branch-2. Thanks [~rkanter] for this handy tool, and [~jlowe] for your reviews. > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6415: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737968#comment-14737968 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #364 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/364/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-tools/hadoop-archive-logs/pom.xml * hadoop-project/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-mapreduce-project/CHANGES.txt * MAPREDUCE-6415.003.patch * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-mapreduce-project/bin/mapred * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738136#comment-14738136 ] Hudson commented on MAPREDUCE-6415: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2313 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2313/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-tools/hadoop-archive-logs/pom.xml * hadoop-project/pom.xml * MAPREDUCE-6415.003.patch * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-mapreduce-project/bin/mapred * hadoop-mapreduce-project/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738148#comment-14738148 ] Hudson commented on MAPREDUCE-6415: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #365 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/365/]) removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738153#comment-14738153 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2290 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2290/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/pom.xml * hadoop-mapreduce-project/bin/mapred * MAPREDUCE-6415.003.patch * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-project/pom.xml * hadoop-mapreduce-project/CHANGES.txt removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738164#comment-14738164 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #351 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/351/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-mapreduce-project/bin/mapred * hadoop-mapreduce-project/CHANGES.txt * hadoop-project/pom.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * hadoop-tools/hadoop-tools-dist/pom.xml * MAPREDUCE-6415.003.patch * hadoop-tools/hadoop-archive-logs/pom.xml removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738092#comment-14738092 ] Hudson commented on MAPREDUCE-6415: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #371 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/371/]) MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) (kasha: rev 119cc75e7ebd723790f6326498383304aba384a2) * hadoop-tools/pom.xml * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogs.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-project/pom.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * MAPREDUCE-6415.003.patch * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogs.java * hadoop-tools/hadoop-tools-dist/pom.xml * hadoop-tools/hadoop-archive-logs/src/test/java/org/apache/hadoop/tools/TestHadoopArchiveLogsRunner.java * hadoop-mapreduce-project/bin/mapred * hadoop-tools/hadoop-archive-logs/src/main/java/org/apache/hadoop/tools/HadoopArchiveLogsRunner.java * hadoop-tools/hadoop-archive-logs/pom.xml removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738108#comment-14738108 ] Hudson commented on MAPREDUCE-6415: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #1103 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1103/]) removing accidental file in MAPREDUCE-6415 (rkanter: rev f15371062f1bbcbb79bf44fd67ec647020d56c69) * MAPREDUCE-6415.003.patch > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.8.0 > > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737270#comment-14737270 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] and [~eepayne] Thank you very much for the comments. Yes. Option 2 will be helpful to cover the existing functionality. And Priority is logically a part of JobStatus, so its good to keep that maximum possible way. However, if AM can get to know the Application Priority, is there any other advantage AM will have other than updating JobStatus. Still its good to do this way as we can avoid 2 RPC calls during every status query. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737264#comment-14737264 ] Karthik Kambatla commented on MAPREDUCE-6415: - LGTM too. Checked with Robert on the findbugs, looks like it was broken before the patch as well. +1 > Create a tool to combine aggregated logs into HAR files > --- > > Key: MAPREDUCE-6415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, > MAPREDUCE-6415.002.patch, MAPREDUCE-6415.002.patch, MAPREDUCE-6415.003.patch, > MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2.002.patch, > MAPREDUCE-6415_branch-2.003.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, > MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, > MAPREDUCE-6415_prelim_002.patch > > > While we wait for YARN-2942 to become viable, it would still be great to > improve the aggregated logs problem. We can write a tool that combines > aggregated log files into a single HAR file per application, which should > solve the too many files and too many blocks problems. See the design > document for details. > See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete
[ https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737484#comment-14737484 ] Jason Lowe commented on MAPREDUCE-5003: --- Thanks for updating the patch, Chang! I tried it out with a sleep job where I manually failed and killed various task attempts and then killed the AM attempt so it would recover. It recovers information for task attempts that had completed, but for task attempts that were active at the time where the AM failed (i.e.: those with a diagnostic of "Killed during application recovery") the log link is broken and the host info is missing. That means we can't figure out where the task attempts were running and can't get to their logs. Other comments on the patch: completedTasksFromPreviousRun is probably not the best name given the code is now placing tasks in that collection that have not completed. Nit: rather than returning from the middle of the TaskImpl.recover function for a running task, I think it would be a bit cleaner to have the code recover a missing taskStatus as taskState=RUNNING and add a RUNNING case to the existing switch statement. > AM recovery should recreate records for attempts that were incomplete > - > > Key: MAPREDUCE-5003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Jason Lowe >Assignee: Chang Li > Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, > MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, > MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch > > > As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task > attempt entries for *all* task attempts launched by the prior app attempt > even if those task attempts did not complete. The attempts would have to be > marked as killed or something similar to indicate it is no longer running. > Having records for the task attempts enables the user to see what nodes were > associated with the attempts and potentially access their logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (MAPREDUCE-6473) Job submission can take a long time during Cluster initialization
[
https://issues.apache.org/jira/browse/MAPREDUCE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-6473 started by Kuhu Shukla.
--
> Job submission can take a long time during Cluster initialization
> -
>
> Key: MAPREDUCE-6473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6473
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>
> During initialization in Cluster.java, the framework provider classes are
> loaded inside a sync block which can considerably increase job submission
> time when the number of submissions are high. The motive is to reduce time
> spent in this sync block safely to improve performance.
> {noformat}
> synchronized (frameworkLoader) {
> for (ClientProtocolProvider provider : frameworkLoader) {
> LOG.debug("Trying ClientProtocolProvider : "
> + provider.getClass().getName());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6473) Job submission can take a long time during Cluster initialization
Kuhu Shukla created MAPREDUCE-6473:
--
Summary: Job submission can take a long time during Cluster
initialization
Key: MAPREDUCE-6473
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6473
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Kuhu Shukla
Assignee: Kuhu Shukla
During initialization in Cluster.java, the framework provider classes are
loaded inside a sync block which can considerably increase job submission time
when the number of submissions are high. The motive is to reduce time spent in
this sync block safely to improve performance.
{noformat}
synchronized (frameworkLoader) {
for (ClientProtocolProvider provider : frameworkLoader) {
LOG.debug("Trying ClientProtocolProvider : "
+ provider.getClass().getName());
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
