[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641019#comment-14641019
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 33s | The applied patch generated 1
new checkstyle issues (total was 28, now 28). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 105m 41s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 142m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.TestMRJobClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745043/0001-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fbd6063 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-jobclient.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/console |
This message was automatically generated.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640790#comment-14640790 ] Jian He commented on MAPREDUCE-5870: patch looks good to me, triggering jenkins Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5870: --- Status: Patch Available (was: Open) Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6394) Speed up Task processing loop in HsTasksBlock#render()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6394: -- Attachment: MAPREDUCE-6394.007.patch - Add two unit tests - Move rawCounter to Counter conversion into single method. Call method in appropriate places - Remove unneeded boolean variable Speed up Task processing loop in HsTasksBlock#render() -- Key: MAPREDUCE-6394 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6394 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Labels: supportability Attachments: MAPREDUCE-6394.001.patch, MAPREDUCE-6394.002.patch, MAPREDUCE-6394.003.patch, MAPREDUCE-6394.004.patch, MAPREDUCE-6394.005.patch, MAPREDUCE-6394.006.patch, MAPREDUCE-6394.007.patch In HsTasksBlock#render(), there is a loop to create a Javascript table which slows down immensely on jobs with a large number of tasks (200k or more). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6438) mapreduce fails with job.jar does not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghvendra Singh updated MAPREDUCE-6438:
Attachment: yarn-site.xml
mapred-site.xml
hdfs-site.xml
core-site.xml
mapreduce fails with job.jar does not exist
---
Key: MAPREDUCE-6438
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6438
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Raghvendra Singh
Attachments: core-site.xml, hdfs-site.xml, mapred-site.xml,
yarn-site.xml
I have a hortonworks distribution(2.2.6.0-2800) of hadoop which runs
mapreduce job based on yarn, and i have a simple map reduce job which reads
compressed data files from hdfs, does some processing over it and then this
data is saved in hbase with bulk load
Here is my program that does it
{code}
final Configuration hadoopConfiguration = new Configuration();
configuration.set(“yarn.resourcemanager.address”, “XX”);
configuration.set(“yarn.resourcemanager.scheduler.address”, “XX”);
configuration.set(mapreduce.framework.name, yarn”);
configuration.set(mapreduce.jobtracker.staging.root.dir,
“”);
final Job job = Job.getInstance(hadoopConfiguration, migration);
job.setJarByClass(BlitzService.class);
job.setMapperClass(DataMigrationMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(KeyValue.class);
job.setReducerClass(DataMigrationReducer.class);
job.setCombinerClass(DataMigrationReducer.class);
HFileOutputFormat2.configureIncrementalLoad(job, hTable);
FileInputFormat.setInputPaths(job, filesToProcess.toArray(new
Path[filesToProcess.size()]));
HFileOutputFormat2.setOutputPath(job, new Path(SOME PATH));
job.waitForCompletion(true);
{code}
This should be a very simple thing to run but i am facing this exception
while running the job
{code}
INFO [2015-07-23 23:53:20,222] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at /172.30.0.147:8032
WARN [2015-07-23 23:53:20,383] org.apache.hadoop.mapreduce.JobSubmitter:
Hadoop command-line option parsing not performed. Implement the Tool
interface and execute your application with ToolRunner to remedy this.
INFO [2015-07-23 23:53:20,492]
org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to
process : 16
INFO [2015-07-23 23:53:20,561] org.apache.hadoop.mapreduce.JobSubmitter:
number of splits:16
INFO [2015-07-23 23:53:20,719] org.apache.hadoop.mapreduce.JobSubmitter:
Submitting tokens for job: job_1437695344326_0002
INFO [2015-07-23 23:53:20,842]
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application
application_1437695344326_0002
INFO [2015-07-23 23:53:20,867] org.apache.hadoop.mapreduce.Job: The url
to track the job:
http://ip-172-30-0-147.us-west-2.compute.internal:8088/proxy/application_1437695344326_0002/
INFO [2015-07-23 23:53:20,868] org.apache.hadoop.mapreduce.Job: Running
job: job_1437695344326_0002
INFO [2015-07-23 23:53:35,994] org.apache.hadoop.mapreduce.Job: Job
job_1437695344326_0002 running in uber mode : false
INFO [2015-07-23 23:53:35,995] org.apache.hadoop.mapreduce.Job: map 0%
reduce 0%
INFO [2015-07-23 23:53:43,053] org.apache.hadoop.mapreduce.Job: Task Id
: attempt_1437695344326_0002_m_01_1000, Status : FAILED
File
file:/tmp/hadoop-yarn/staging/root/.staging/job_1437695344326_0002/job.jar
does not exist
java.io.FileNotFoundException: File
file:/tmp/hadoop-yarn/staging/root/.staging/job_1437695344326_0002/job.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:608)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:821)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:598)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:414)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
[jira] [Created] (MAPREDUCE-6438) mapreduce fails with job.jar does not exist
Raghvendra Singh created MAPREDUCE-6438:
---
Summary: mapreduce fails with job.jar does not exist
Key: MAPREDUCE-6438
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6438
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Raghvendra Singh
I have a hortonworks distribution(2.2.6.0-2800) of hadoop which runs mapreduce
job based on yarn, and i have a simple map reduce job which reads compressed
data files from hdfs, does some processing over it and then this data is saved
in hbase with bulk load
Here is my program that does it
{code}
final Configuration hadoopConfiguration = new Configuration();
configuration.set(“yarn.resourcemanager.address”, “XX”);
configuration.set(“yarn.resourcemanager.scheduler.address”, “XX”);
configuration.set(mapreduce.framework.name, yarn”);
configuration.set(mapreduce.jobtracker.staging.root.dir, “”);
final Job job = Job.getInstance(hadoopConfiguration, migration);
job.setJarByClass(BlitzService.class);
job.setMapperClass(DataMigrationMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(KeyValue.class);
job.setReducerClass(DataMigrationReducer.class);
job.setCombinerClass(DataMigrationReducer.class);
HFileOutputFormat2.configureIncrementalLoad(job, hTable);
FileInputFormat.setInputPaths(job, filesToProcess.toArray(new
Path[filesToProcess.size()]));
HFileOutputFormat2.setOutputPath(job, new Path(SOME PATH));
job.waitForCompletion(true);
{code}
This should be a very simple thing to run but i am facing this exception while
running the job
{code}
INFO [2015-07-23 23:53:20,222] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at /172.30.0.147:8032
WARN [2015-07-23 23:53:20,383] org.apache.hadoop.mapreduce.JobSubmitter:
Hadoop command-line option parsing not performed. Implement the Tool interface
and execute your application with ToolRunner to remedy this.
INFO [2015-07-23 23:53:20,492]
org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to
process : 16
INFO [2015-07-23 23:53:20,561] org.apache.hadoop.mapreduce.JobSubmitter:
number of splits:16
INFO [2015-07-23 23:53:20,719] org.apache.hadoop.mapreduce.JobSubmitter:
Submitting tokens for job: job_1437695344326_0002
INFO [2015-07-23 23:53:20,842]
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application
application_1437695344326_0002
INFO [2015-07-23 23:53:20,867] org.apache.hadoop.mapreduce.Job: The url to
track the job:
http://ip-172-30-0-147.us-west-2.compute.internal:8088/proxy/application_1437695344326_0002/
INFO [2015-07-23 23:53:20,868] org.apache.hadoop.mapreduce.Job: Running
job: job_1437695344326_0002
INFO [2015-07-23 23:53:35,994] org.apache.hadoop.mapreduce.Job: Job
job_1437695344326_0002 running in uber mode : false
INFO [2015-07-23 23:53:35,995] org.apache.hadoop.mapreduce.Job: map 0%
reduce 0%
INFO [2015-07-23 23:53:43,053] org.apache.hadoop.mapreduce.Job: Task Id :
attempt_1437695344326_0002_m_01_1000, Status : FAILED
File
file:/tmp/hadoop-yarn/staging/root/.staging/job_1437695344326_0002/job.jar does
not exist
java.io.FileNotFoundException: File
file:/tmp/hadoop-yarn/staging/root/.staging/job_1437695344326_0002/job.jar does
not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:608)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:821)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:598)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:414)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641134#comment-14641134 ] Jian He commented on MAPREDUCE-5870: [~sunilg], could you check if the test failure is related ? Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641358#comment-14641358 ] Sunil G commented on MAPREDUCE-5870: Thank you [~jianhe] for review and sharing the comments. I have looked into the test failures and analyzed the problem. It seems we still have implementat Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6394) Speed up Task processing loop in HsTasksBlock#render()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641282#comment-14641282
]
Hadoop QA commented on MAPREDUCE-6394:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 37s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 16s | The applied patch generated 2
new checkstyle issues (total was 98, now 100). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 18s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 4s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 5m 54s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 57m 36s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12747124/MAPREDUCE-6394.007.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / ff9c13e |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5911/console |
This message was automatically generated.
Speed up Task processing loop in HsTasksBlock#render()
--
Key: MAPREDUCE-6394
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6394
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: jobhistoryserver
Affects Versions: 2.7.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Labels: supportability
Attachments: MAPREDUCE-6394.001.patch, MAPREDUCE-6394.002.patch,
MAPREDUCE-6394.003.patch, MAPREDUCE-6394.004.patch, MAPREDUCE-6394.005.patch,
MAPREDUCE-6394.006.patch, MAPREDUCE-6394.007.patch
In HsTasksBlock#render(), there is a loop to create a Javascript table which
slows down immensely on jobs with a large number of tasks (200k or more).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641360#comment-14641360
]
Sunil G commented on MAPREDUCE-5870:
Thank you [~jianhe] for review and sharing the comments.
I have looked into the test failures and analyzed the problem. It seems we
still have implementation of old MRV1 priority in code.
{code}
@InterfaceAudience.Public
@InterfaceStability.Stable
public enum JobPriority {
VERY_HIGH,
HIGH,
NORMAL,
LOW,
VERY_LOW;
}
{code}
This enum is used in many places in code. Also we have some stable apis in
JobConf related to JobPriority.
{code}
public void setJobPriority(JobPriority prio)
public JobPriority getJobPriority()
{code}
Do we still need to keep any backward compatibility support to this? If so, we
can do a type convert for this macro and set priority accordingly in
YarnRunner. Otherwise we may need to remove some of unused test cases from MR
related to old way of setting priority.
Kindly advice.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
