[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025627#comment-15025627
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8881 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8881/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
* hadoop-mapreduce-project/CHANGES.txt
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025668#comment-15025668
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #718 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/718/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025851#comment-15025851
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #728 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/728/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025890#comment-15025890
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2660 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2660/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026036#comment-15026036
]
Sunil G commented on MAPREDUCE-5870:
Thank you very much [~jlowe] for the reviews, commit and the support. And thank
you [~jianhe], [~devaraj.k] and [~eepayne] for the reviews !
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026142#comment-15026142
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #2577 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2577/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026197#comment-15026197
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #640 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/640/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025787#comment-15025787
]
Hudson commented on MAPREDUCE-5870:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #1450 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1450/])
MAPREDUCE-5870. Support for passing Job priority through Application (jlowe:
rev f634505d48d97e4d461980d68a0cbdf87223646d)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobStatus.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobPriority.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobPriority.java
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Fix For: 2.8.0
>
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025507#comment-15025507
]
Jason Lowe commented on MAPREDUCE-5870:
---
Thanks for updating the patch, Sunil! +1, agree that unit test failures and
java warnings are unrelated to this patch. Committing this.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020392#comment-15020392
]
Sunil G commented on MAPREDUCE-5870:
MAPREDUCE-6555 is filed by Varun for tracking {{TestMRAppMaster}} failures.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020354#comment-15020354
]
Sunil G commented on MAPREDUCE-5870:
Findbugs warnings are unrelated. Few javac warnings are populated due to
deprecation which is not introduced by this one. I will check why this happened.
{noformat}
DistributedCache in org.apache.hadoop.mapreduce.filecache has been deprecated
{noformat}
I will file a ticket for {{TestMRAppMaster.testMRAppMasterMidLock}} failure.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, 0012-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019031#comment-15019031
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 25s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 2s {color}
| {color:red} hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with
JDK v1.8.0_66 generated 21 new issues (was 360, now 360). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 32s
{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_85 with JDK v1.7.0_85
generated 21 new issues (was 365, now 365). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 385, now 381).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 2s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 129m 23s
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013931#comment-15013931
]
Sunil G commented on MAPREDUCE-5870:
Sure [~jlowe]. I will upload a patch addressing the same.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15011905#comment-15011905
]
Jason Lowe commented on MAPREDUCE-5870:
---
It would be nice to cleanup most of the checkstyle nits, as there are a lot of
switch statements without default clauses that were added. I realize
semantically we're covered since the fall-through case returns some default
value or throws, but we can trivially remove the warning by having an explicit
default clause that explicitly breaks or move the fallthrough code into that
default case.
Other than that latest patch looks good to me.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Priority support in MapReduce.
> 1. Job Priority can be set from client side as below (Configuration and api).
> - JobConf.getJobPriority() and
> JobConf.setJobPriority(JobPriority priority)
> - We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
> 2. CLI changes to support {{--set-priority}}. Run time change of JobPriority
> is also to be handled.
> 3. Change {{Job}} to have the support for {{setPriority}} and {{getPriority}}
> (getter is handled with JobStatus)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008479#comment-15008479
]
Sunil G commented on MAPREDUCE-5870:
Test case failures are not related. {{TestJobImpl}} is passing locally. Others
are failing w/o the patch also. I will see whether any test failure tickets are
raised, if not will do the same.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, 0010-MAPREDUCE-5870.patch,
> 0011-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007416#comment-15007416
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 9 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 385, now 388).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 45s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 1s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 26s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 17s
{color} |
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008171#comment-15008171
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 38s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 19s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 9 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 385, now 388).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 6s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 20s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 46s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 59s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_85.
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000638#comment-15000638 ] Sunil G commented on MAPREDUCE-5870: Thanks [~jlowe]. Yes, it's a race condition. Job priority change is done from client, then it's getting updated in RM. Once AM heartbeat is up, latest priority is updated to AM. Then with a client JobStatus query, client will get this. So a sleep is needed at client side till this cycle is done, which keeps varying. I ll look into the timeout again and will tune better in test case. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, > 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000567#comment-15000567
]
Jason Lowe commented on MAPREDUCE-5870:
---
I still think the TestUberAM failure is related to this patch. From the latest
Jenkins run log at
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6118/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
{noformat}
testJobWithChangePriority(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
elapsed: 21.508 sec <<< FAILURE!
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobWithChangePriority(TestMRJobs.java:276)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
The failing test was added with this patch. Since it passes on JDK7, maybe the
test has a race condition?
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998906#comment-14998906 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] I have updated the patch correcting test failures, I will make changes for LocalJobRunner in separate ticket as you suggested earlier since it was already throwing exception. There were some failures in JDK 1.8, but locally it looks fine. I will re trigger jenkins again. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, > 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991564#comment-14991564
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 9 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 381, now 384).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 7s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 4s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 121m 0s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 19s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.7.0_79. {color} |
|
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991825#comment-14991825
]
Sunil G commented on MAPREDUCE-5870:
Test case of {{TestUberAM}} is passing locally and also its fine for JDK 1.7.
But Jenkins reports failure for JDK 1.8. I will check jenkins again to see the
results.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch,
> 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch,
> 0009-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989960#comment-14989960 ] Jason Lowe commented on MAPREDUCE-5870: --- TestUberAM failure appears to be related, and I'm not sure about the others. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, 0008-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984442#comment-14984442
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 10 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 380, now 385).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 59s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 33s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 49s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 8s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 0s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 24s
{color} | {color:red} hadoop-mapreduce-client-jobclient in
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982600#comment-14982600
]
Hadoop QA commented on MAPREDUCE-5870:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 22 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 380, now 397).
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 40s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 22s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed with
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 0s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 18s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed
with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 34s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 239m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982632#comment-14982632 ] Sunil G commented on MAPREDUCE-5870: Test case failures looks related. I will fix them in the next patch. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, > 0007-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746801#comment-14746801 ] Sunil G commented on MAPREDUCE-5870: Thank you [~jlowe] for the detailed information. I will raise a YARN ticket which will update the AllocatedResponse to carry application priority. Also with another MR ticket, i will update JobStatus with priority which came from AllocatedResponse. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745506#comment-14745506 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] [~jianhe] [~eepayne] I thought of bringing up a point we discussed earlier as I have one more concern. To keep priority in JobStatus, and also to contact only AM (not RM) to get JobStatus, we had discussed on updating priority to AM from RM through heartbeat. However, other than updating JobStatus, I am not finding any other good useful reason for AM to know its own priority in RM. Do you see any more advantages for AM knowing its priority(may be other than MapReduce AM also). > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745511#comment-14745511 ] Jason Lowe commented on MAPREDUCE-5870: --- I too struggle with whether it's worth going through all this trouble. I honestly don't know how much the MRv1 priority API was used and therefore the impact of completely abandoning it. If we are going to keep supporting the MRv1 priority API, what are your thoughts about handling the reporting of app priority via JobStatus from the AM? I really don't want YARNRunner hitting the RM for every status update, as that will put a lot more RPC load on the RM than we have today. But the alternative means we either need to have the AM report a delayed view of the priority (e.g.: send priority in heartbeat response, AM may be out-of-sync by up to one heartbeat interval), or we have to deprecate that part of the MRv1 API. I think I'm OK with having the priority reported in the AM heartbeat. It will be somewhat delayed with reality when it changes, but it should be relatively rare for it to be out of sync more than a few seconds. The priority would have to have changed while the AM is having difficulty contacting the RM. If that persists too long either the RM will expire the AM and kill it or the AM will give up on the RM and exit. Having the AM know the priority could also help with app frameworks that want to report this in their AM-provided UI. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744709#comment-14744709 ] Jian He commented on MAPREDUCE-5870: I see, I'm ok to keep it supporting the enum as that's what I originally thought. Just want to bring this up. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743954#comment-14743954 ] Jason Lowe commented on MAPREDUCE-5870: --- I honestly don't know how much the priority APIs were used from MRv1. However given that there were priority APIs in MRv1 that's the most logical thing people migrating from Hadoop 1 will expect to use when they hear priorities are finally working in Hadoop 2. If we decide to deprecate the old MRv1 priority API, should we avoid using mapreduce.job.priority (was mapred.job.priority)? Wondering if jobs that were working in Hadoop 2 suddenly start failing during job submit because someone was specifying something for that property that was always ignored and now the new code barfs on any non-integer found there. However I'm not excited about the confusion of having two similar properties to set the job priority, one that works and one that does not. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743154#comment-14743154 ] Jian He commented on MAPREDUCE-5870: I earlier thought we can keep backward compatible with the enum priority, but now am thinking the value of doing this. This does bring extra complexity to support both. [~jlowe], do you know if there are many apps from MR1 are actually expecting this enum based priority to work ? Since priority is never supported since hadoop 2 for such a long time, I'm thinking if we can deprecate the old API and claim only support integers to be simple and clear. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741059#comment-14741059 ] Sunil G commented on MAPREDUCE-5870: Yes. I thought of not updating JobStatus for priority from RM here for now. Once we do in RM, I will update priority to JobStatus from updated JobReport. Looping [~jianhe] > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739298#comment-14739298 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] [~eepayne] [~jianhe] As discussed if option-2 is fine, I will raise a separate ticket in YARN to handle RM-AM update of priority (through heartbeat). And I will separate the JobStatus priority update from this ticket for now. And once YARN is done, we can handle JobStatus priority update in correct way. Could you share your opinion on same. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739504#comment-14739504
]
Eric Payne commented on MAPREDUCE-5870:
---
{quote}
As discussed if option-2 is fine, I will raise a separate ticket in YARN to
handle RM-AM update of priority (through heartbeat).
And I will separate the JobStatus priority update from this ticket for now.
{quote}
Thanks, [~sunilg]. If I understand correctly, you will update the patch for
this JIRA so that the AM will not query the RM for it's job priority, and then
in another JIRA, mkae the change to have the RM tell the AM its priority as
part a heartbeat ack. Is that correct? I just want to make sure that this JIRA
doesn't add that extra load on the RM for an AM JobStatus query.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736993#comment-14736993
]
Sunil G commented on MAPREDUCE-5870:
Thank you [~jlowe] for sharing the comments.
As discussed offline, I had doubts on couple of points here.
bq.We are already returning the job priority via getJobStatus
As I see, we are invoking {{YARNRunner#getJobStatus}} to get the job status
while we try to call {{Job#getPriority}}.
And in {{ClientServiceDelegate#getJobStatus}}, *JobStatus* is created from
*JobReport* (from AM).
{code}
public JobStatus getJobStatus(JobID oldJobID) throws IOException {
org.apache.hadoop.mapreduce.v2.api.records.JobId jobId =
TypeConverter.toYarn(oldJobID);
GetJobReportRequest request =
recordFactory.newRecordInstance(GetJobReportRequest.class);
request.setJobId(jobId);
JobReport report = ((GetJobReportResponse) invoke("getJobReport",
GetJobReportRequest.class, request)).getJobReport();
...
...
jobStatus = TypeConverter.fromYarn(report, url);
...
{code}
As this JobReport doesnt have priority information, current code in
{{TypeConverter#fromYarn}} is to hardcode the status.
{code}
public static JobStatus fromYarn(JobReport jobreport, String trackingUrl) {
JobPriority jobPriority = JobPriority.NORMAL;
JobStatus jobStatus = new org.apache.hadoop.mapred.JobStatus(
fromYarn(jobreport.getJobId()), jobreport.getSetupProgress(), jobreport
{code}
AppReport has the information about the priority of the application. So a
second call has to be made to fetch the same from {{YarnRunner#getJobStatus}}.
And it can be set to JobStatus separately.
Kindly help to share your thoughts here if I am missing something important.
bq.Why was the call to testChangingJobPriority removed from TestMRJobClient?
{{TestMRJobClient#testChangingJobPriority}} was invoked after job was
completed. And YARN throws excpetion for change priority application if its not
RUNNING or ACCEPTED. As updated offline, I agree that YARN need not have to
throw excpetion if the app is completed, rather it can be skipped. I will file
a ticket in YARN to fix the same. Thoughts?
For LocalJobRunner exception case, I will file another ticket to fix the
exception and to support correct priority.
Also I will add all possible test cases for Job and JobConf to see all possible
integer and enums are set/get correctly.
Thank you for the support.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737035#comment-14737035 ] Jason Lowe commented on MAPREDUCE-5870: --- bq. AppReport has the information about the priority of the application. So a second call has to be made to fetch the same from YarnRunner#getJobStatus. And it can be set to JobStatus separately. The problem is that the ApplicationMaster doesn't actually know what its job priority actually is, as that's a concept tracked by the RM not the AM. So to report the priority we'd have to make a separate call to the RM to determine the priority and meld that into the rest of the information returned for the job status. I'm not thrilled with the thought of adding an additional call to the RM for every time a client wants to get the job status, especially for a field that most clients won't need. Clients often bang on the job status to see when a job is done, and that means they'll start banging on the RM as well if we make the delegate start hitting both the AM and the RM to generate the JobStatus. There's only two ways I can currently think of to get around that. Either we need to obsolete the priority field from the job status and provide a separate call to get the priority when the client really wants to know it, or we have to find a way for the AM to know its job priority so it can return it in its JobStatus responses. For the latter we could have the RM send it down in heartbeat responses, but there would be a delay between when a client updates the priority and the AM reports the updated value. We could have the AM proactively ask the RM what its priority is, but then we're back to banging on the RM for every job status call (only this time from AMs instead of the client, but same number of RM calls overall). Any other thoughts on what we should do here? I'm not thrilled about the thought of ditching the priority in the JobStatus field. However it's been worthless ever since we went to YARN (i.e.: broken since 0.23 and 2.x), so it's tempting to declare it officially obsolete (as it in practice already is) and provide a different API to retrieve it. bq. And YARN throws excpetion for change priority application if its not RUNNING or ACCEPTED. Yes, this to me is a bug. We don't throw exceptions when clients try to kill apps that have terminated, and similarly we shouldn't throw exceptions if clients try to change the priority of terminated apps. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737076#comment-14737076
]
Eric Payne commented on MAPREDUCE-5870:
---
{quote}
There's only two ways I can currently think of to get around that. Either we
need to obsolete the priority field from the job status and provide a separate
call to get the priority when the client really wants to know it, or we have to
find a way for the AM to know its job priority so it can return it in its
JobStatus responses. For the latter we could have the RM send it down in
heartbeat responses, but there would be a delay between when a client updates
the priority and the AM reports the updated value.
{quote}
[~jlowe] and [~sunilg],
I would vote for the second option. That is, update the priority in the RM and
then tell the AM about it when it hearbeats in. I think it would be fine to
have the short delay between when the priority was updated and the AM knows
about it. I can't foresee a use case where the client would atomically need to
know about the change. The one thing that might cause confusion is if the
client set the priority and immediately read the priority which might still be
the old value. This may cause confusion, but as long as this behavior is well
documented it should be fine.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737270#comment-14737270 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] and [~eepayne] Thank you very much for the comments. Yes. Option 2 will be helpful to cover the existing functionality. And Priority is logically a part of JobStatus, so its good to keep that maximum possible way. However, if AM can get to know the Application Priority, is there any other advantage AM will have other than updating JobStatus. Still its good to do this way as we can avoid 2 RPC calls during every status query. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735105#comment-14735105 ] Jason Lowe commented on MAPREDUCE-5870: --- Thanks for updating the patch! We cannot change ClientProtocol.setJobPriority to take an integer rather than a String. That's a backwards-incompatible change. Why are we adding ClientProtocol.getJobPriority? We are already returning the job priority via getJobStatus, so I'm a little confused on why we are extending the protocol to get information the client can already retrieve. Does it make more sense to have LocalJobRunner.getJobPriority return a normal or default priority? Returning null seems disruptive, as priorities really don't matter in the local job mode scenario. In a similar sense, I'm wondering if we should have LocalJobRunner.setJobPriority simply track or ignore the priority rather than explode. Since it already exploded for setJobPriority before we can tackle that in a separate JIRA if desired. Why does JobConf.setPriorityAsInteger take a boxed type rather than a normal int and call Integer.toString? I think the new TypeConverter functions are a little too generically named. It's a bit dangerous to have a toYarn that takes something as generic as a String and returns an int -- it could easily me misused in other contexts that have nothing to do with job priorities. I'd name these fromYarnPriority, toYarnPriority, etc. so there's no chance of accidental collision. convertPriorityToInteger needs to handle the case where the string is itself an integer, otherwise setJobPriorityAsInteger followed by getJobPriorityAsInteger doesn't work as expected. Trying to do JobPriority.valueOf on a string that may not contain a valid enum is going to throw an IllegalArgumentException. Please add unit tests for the various get/set as integer scenarios on both Job and JobConf to make sure we're handling them properly. Why was the call to testChangingJobPriority removed from TestMRJobClient? The function is still there, although in practice is never called. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, > 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729330#comment-14729330
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 10m 8s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 34s | The applied patch generated 8
new checkstyle issues (total was 367, now 371). |
| {color:red}-1{color} | whitespace | 0m 9s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 49s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 96m 38s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 143m 43s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.TestValueIterReset |
| | hadoop.mapred.TestYARNRunner |
| | hadoop.mapreduce.TestChild |
| | hadoop.mapreduce.TestMapReduceLazyOutput |
| | hadoop.mapreduce.TestMRJobClient |
| | hadoop.mapred.TestNetworkedJob |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12753984/0006-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9a87f81 |
| javadoc |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5969/console |
This message was automatically generated.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14728017#comment-14728017
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 32s | The applied patch generated
49 new checkstyle issues (total was 367, now 412). |
| {color:red}-1{color} | whitespace | 0m 8s | The patch has 9 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 22s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 70m 53s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 116m 35s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestReduceFetch |
| | hadoop.mapred.TestNetworkedJob |
| | hadoop.mapred.TestJobName |
| | hadoop.mapred.TestLazyOutput |
| | hadoop.mapreduce.TestMRJobClient |
| Timed out tests | org.apache.hadoop.mapred.TestMiniMRClasspath |
| | org.apache.hadoop.mapred.TestLocalMRNotification |
| | org.apache.hadoop.mapred.TestLineRecordReaderJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12753821/0005-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7d6687f |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5968/console |
This message was automatically generated.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch,
> 0005-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725031#comment-14725031 ] Sunil G commented on MAPREDUCE-5870: Thank you [~jlowe] for the detailed comments. I will work on these items and will update a patch shortly. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723469#comment-14723469 ] Jason Lowe commented on MAPREDUCE-5870: --- Thanks for updating the patch, Sunil! I'm not sure why we need an updateJobPriority that's separate from the existing setJobPriority call that's already there. Seems like we just need to put the code for updateApplicationPriority into ResourceMgrDelegate.setJobPriority which currently does nothing. Why does setJobPriority return DEFAULT if the priority appears to be a negative number but returns UNDEFINED_PRIORITY if the number is not negative? I don't recall negative priority values being special. We need to translate equivalent integer priorities to the corresponding enum. For example, if someone sets the priority as 5 then getJobPriority should return VERY_HIGH. Otherwise when someone sets a priority via enum they will get back UNDEFINED_PRIORITY instead of what they set which is inconsistent. Job.setPriority should preserve the enum name in the conf for better backwards compatibility. I think we should pass the Job.setPriority argument straight through JobConf.setJobPriority if the enum isn't UNDEFINED_PRIORITY. TypeConverter needs to handle the new DEFAULT priority as zero. Nit: Please add comments to the two new JobPriority enums explaining their usage. The handling of priority from JobStatus also needs to be updated. Currently the code is hardwired to return a NORMAL priority for a running job. See the way TypeConverter handles an application report and builds up a job status. There should be some unit tests. Minimally there should be tests to check some of the things we've discussed wrt. consistency of priority get and set. For example, setting a priority enum should return that enum on get for the old priority enums, etc. There should also be a test that verifies we are passing the priority in the app submission context appropriately and calling the appropriate API to update job priority when job.setPriority is called on a running job. Finally there should be tests to verify we are marshalling a YARN application report properly wrt. priority handling. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711373#comment-14711373
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 59s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 31s | The applied patch generated
26 new checkstyle issues (total was 255, now 277). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 21s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:red}-1{color} | mapreduce tests | 1m 41s | Tests failed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 108m 1s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 153m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestJobConf |
| | hadoop.mapreduce.TestMRJobClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752221/0004-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/console |
This message was automatically generated.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703060#comment-14703060 ] Jason Lowe commented on MAPREDUCE-5870: --- bq. But now YARN has the capability to take default value, if priority is not set from client side. That's exactly what I intended for the new DEFAULT enum to indicate. If set on the client side then the job submitter shouldy avoid setting any job priority and let the cluster set an automatic value. In other words, this specific enum value should be treated as if there was no conf property set. bq. So I can make 0 as DEFAULT (new enum) Note that we should not treat a priority of 0 as DEFAULT. As I mentioned above, DEFAULT means there is no explicit priority set, so no integer priority value should map to the DEFAULT enum. bq. I agree with your idea of having a new enum which can be returned when any other integer is been used a priority other than standard enum. My main complaint with that approach is that it's not symmetric. That enum value cannot be set by clients, and it should be treated as an error since we cannot know which integer priority corresponds. But at least it keeps the getJobPriority call from crashing, and I don't know of a better approach offhand. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703032#comment-14703032
]
Sunil G commented on MAPREDUCE-5870:
Thanks [~jlowe] for sharing the inputs.
Yes, as you mentioned it wont be as perfect as we cant map some undefined
numerical constants to mapred defined enums.
Few points:
- As you have mentioned, we could add a DEFAULT enum to indicate a case where
no priority is set from client side. But now YARN has the capability to take
default value, if priority is not set from client side. This can be Queue
default or Cluster default (if Queue doesnt have any default or queue doesnt
exist). And mapred cannot know this. So could we add a DEFAULT enum with 0 as
base value for now.
- There is another problem when we do integer mapping.
{code}
public enum JobPriority {
VERY_HIGH,
HIGH,
NORMAL,
LOW,
VERY_LOW;
}
{code}
We wont be able to take *ordinal* here because of the current order. Hence
{TypeConver#toYarn}} can help in mapping this enum to a YARN model of integer
priority (preferably from 1 to 5). So I can make 0 as DEFAULT (new enum)
- I agree with your idea of having a new enum which can be returned when any
other integer is been used a priority other than standard enum. Eventhough its
not much helpful as its not giving exact priority value, I feel we can also
have a new api {{public int getJobPriorityAsInteger}} along with existing one
for more clarity in such cases.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703109#comment-14703109 ] Sunil G commented on MAPREDUCE-5870: bq.As I mentioned above, DEFAULT means there is no explicit priority set, so no integer priority value should map to the DEFAULT enum Yes. That's correct, if we set some other value, it will give a wrong meaning as it can also be a valid priority in real case. I will use plain enum without any mapping as such. bq.But at least it keeps the getJobPriority call from crashing Yes, its like handling a default case. I will try giving a more meaning name for that ENUM, so users can try new api to get exact integer mapping value for priority atleast. As YARN-4014 is getting in, I will handle *mapred* cli command also to set priority at run time in same ticket. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703789#comment-14703789 ] Eric Payne commented on MAPREDUCE-5870: --- [~sunilg], bq. As YARN-4014 is getting in, I will handle mapred cli command also to set priority at run time in same ticket. Are you saying that you will be combining pieces of this JIRA (MAPREDUCE-5870) with YARN-4014? Can you please clarify which pieces? Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704296#comment-14704296
]
Sunil G commented on MAPREDUCE-5870:
HI [~eepayne]
In Job class, we have an api called {{setJobPriority}} and it invokes
setJobPriority api in ClientProtocol if job is running. And this api is dummy
for now.
Job.java
{code}
ensureState(JobState.RUNNING);
final JobPriority tmpPriority = priority;
ugi.doAs(new PrivilegedExceptionActionObject() {
@Override
public Object run() throws IOException, InterruptedException {
cluster.getClient().setJobPriority(getJobID(),
tmpPriority.toString());
return null;
}
});
{code}
ResourceMgDelegate.java
{code}
public void setJobPriority(JobID arg0, String arg1) throws IOException,
InterruptedException {
return;
}
{code}
YARN-4014 introduces new api in {{ClientProtocol}} to set priority of an
application at runtime. So we can use that api instead of current dummy one.
Existing cli is {{mapred -set-priority values}}
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702081#comment-14702081 ] Jason Lowe commented on MAPREDUCE-5870: --- Thanks for the patch, Sunil. Looks good in general, but I think we need to decide what to do about JobConf.getJobPriority. Currently it will return NORMAL if no priority was set but yet the job will _not_ submit with that priority in practice. In addition JobConf.getJobPriority will crash if the priority wasn't set to one of the priority enum strings. Not sure there's a great fix for these. We could add a new enum, like DEFAULT or QUEUE_DEFAULT, which indicates the job priority has not been specified and we could at least do something symmetrical there. A client can see that enum returned or even set it explicitly, and the submit code can treat that enum as if no priority was set. The arbitrary priority value mapping to a mapred priority is trickier. We could add an enum to indicate the priority doesn't map to one of the other standard values, but that would only provide half of the picture. If a user set that priority explicitly then we wouldn't know what priority number corresponds. However we could at least map the numerical values to the standard priority enums so we're symmetrical with the toYarn behavior. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663327#comment-14663327 ] Sunil G commented on MAPREDUCE-5870: Thank you very much [~eepayne] for helping to verify the patch. I will update the patch by adding the import. This patch is also now depends on YARN-3887 and YARN-4014 as *mapred* cli has a command to set the priority of an application at runtime. I will rebase as those tickets are committed. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663133#comment-14663133
]
Eric Payne commented on MAPREDUCE-5870:
---
Hi [~sunilg],
One thing I forgot to mention was that I had to add the following line to
{{YARNRunner.java}} to get it to compile.
{code}
import org.apache.hadoop.yarn.api.records.Priority;
{code}
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662064#comment-14662064
]
Eric Payne commented on MAPREDUCE-5870:
---
[~sunilg], for what it's worth, I have downloaded the latest patch (version
003) and tested and verified it in conjunction with the changes that were made
for YARN-2003.
I performed the following sleep jobs with 10 tasks each. My one-node cluster
can run 5 containers at once.
- I submit sleep job1 to the default queue, setting
{{-Dmapreduce.job.priority=LOW}}
- Job1 starts running 5 containers and has 5 tasks pending.
- I submit sleep job2 to the default queue, setting
{{-Dmapreduce.job.priority=HIGH}}
- All 10 job2 tasks are pending.
- Once tasks from job1 complete, job2 gets the containers. Although job1 has 5
tasks pending, the number of running tasks for job1 remains 0 until job2 has no
more pending tasks and job2's running tasks begin to complete.
- At that point, job1's tasks begin again to receive containers.
I also verified that you can specify {{-Dmapreduce.job.priority=_number_}}, and
the container allocations go to the higher numbered jobs.
Finally, I verified that if you make the priority higher than the cluster max,
it silently sets the job priority to cluster max.
So, the bottom line is LGTM :-)
+1
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643072#comment-14643072
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 55s | The applied patch generated 1
new checkstyle issues (total was 33, now 34). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 109m 10s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 149m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12747341/0003-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2196e39 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5913/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5913/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5913/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5913/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5913/console |
This message was automatically generated.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
0003-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641903#comment-14641903
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 55s | The applied patch generated
11 new checkstyle issues (total was 33, now 44). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 0s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 110m 36s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 150m 54s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12747222/0002-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 156f24e |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5912/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5912/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5912/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5912/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5912/console |
This message was automatically generated.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642080#comment-14642080
]
Jian He commented on MAPREDUCE-5870:
[~sunilg], thanks for updating,
- should we use the JobConf.getJobPriority API so that it can accept the
current CLI specified priority too ?
{code}
String jobPriority = jobConf.get(MRJobConfig.PRIORITY);
{code}
- to simplify a little bit,
{code}
int iPriority = TypeConverter.toYarn(jobPriority);
// If the given input not a JobPriority enum, verify whether its an
// integer.
if (0 == iPriority) {
iPriority = Integer.parseInt(jobPriority);
}
{code}
we can do something like
{code}
try {
iPriority = TypeConverter.toYarn(jobPriority);
} catch (IllegalArgumentException exception) {
iPriority = Integer.parseInt(jobPriority);
}
{code}
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641019#comment-14641019
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 33s | The applied patch generated 1
new checkstyle issues (total was 28, now 28). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 105m 41s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 142m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.TestMRJobClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745043/0001-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fbd6063 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-jobclient.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5910/console |
This message was automatically generated.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640790#comment-14640790 ] Jian He commented on MAPREDUCE-5870: patch looks good to me, triggering jenkins Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641134#comment-14641134 ] Jian He commented on MAPREDUCE-5870: [~sunilg], could you check if the test failure is related ? Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641358#comment-14641358 ] Sunil G commented on MAPREDUCE-5870: Thank you [~jianhe] for review and sharing the comments. I have looked into the test failures and analyzed the problem. It seems we still have implementat Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641360#comment-14641360
]
Sunil G commented on MAPREDUCE-5870:
Thank you [~jianhe] for review and sharing the comments.
I have looked into the test failures and analyzed the problem. It seems we
still have implementation of old MRV1 priority in code.
{code}
@InterfaceAudience.Public
@InterfaceStability.Stable
public enum JobPriority {
VERY_HIGH,
HIGH,
NORMAL,
LOW,
VERY_LOW;
}
{code}
This enum is used in many places in code. Also we have some stable apis in
JobConf related to JobPriority.
{code}
public void setJobPriority(JobPriority prio)
public JobPriority getJobPriority()
{code}
Do we still need to keep any backward compatibility support to this? If so, we
can do a type convert for this macro and set priority accordingly in
YarnRunner. Otherwise we may need to remove some of unused test cases from MR
related to old way of setting priority.
Kindly advice.
Support for passing Job priority through Application Submission Context in
Mapreduce Side
-
Key: MAPREDUCE-5870
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch
Job Prioirty can be set from client side as below [Configuration and api].
a. JobConf.getJobPriority() and
Job.setPriority(JobPriority priority)
b. We can also use configuration
mapreduce.job.priority.
Now this Job priority can be passed in Application Submission
context from Client side.
Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14639191#comment-14639191 ] Sunil G commented on MAPREDUCE-5870: Hi [~jlowe] [~devaraj.k] Could you please take a look on this patch. YARN-2003 is committed in trunk, so with this change we can submit an MR job with app priority. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: 0001-MAPREDUCE-5870.patch, Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986531#comment-13986531 ] Sunil G commented on MAPREDUCE-5870: Thank you Jason and Devaraj for the comments. I will upload the patch with test cases as mentioned. Also in the parent jira, there are some discussions happening on same. So based on that, the patch will be uploaded. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985729#comment-13985729 ] Devaraj K commented on MAPREDUCE-5870: -- Please refer TestTypeConverter.java for adding tests for this. Support for passing Job priority through Application Submission Context in Mapreduce Side - Key: MAPREDUCE-5870 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Sunil G Assignee: Sunil G Attachments: Yarn-2002.1.patch Job Prioirty can be set from client side as below [Configuration and api]. a. JobConf.getJobPriority() and Job.setPriority(JobPriority priority) b. We can also use configuration mapreduce.job.priority. Now this Job priority can be passed in Application Submission context from Client side. Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.2#6252)
