[jira] [Created] (MAPREDUCE-7365) AppMaster register UAM can lost application priority in subCluster
Yao Guangdong created MAPREDUCE-7365:
Summary: AppMaster register UAM can lost application priority in
subCluster
Key: MAPREDUCE-7365
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7365
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, nodemanager, resourcemanager
Affects Versions: 3.3.0, 3.1.0, 3.0.0, 2.9.0
Reporter: Yao Guangdong
Fix For: 3.3.2
AppMaster register uam to subCluster can lost application priority in yarn
federation cluster.Which make the subCluster's RM allocate resouce to
application use default priority regardless of the application's real priority.
By analyzing the code, I found that the appMaster submitApplication to
subCluster did't set the priority to ApplicationSubmissionContext. As follows:
{code:java}
private void submitUnmanagedApp(ApplicationId appId)
throws YarnException, IOException {
SubmitApplicationRequest submitRequest =
this.recordFactory.newRecordInstance(SubmitApplicationRequest.class);
ApplicationSubmissionContext context = this.recordFactory
.newRecordInstance(ApplicationSubmissionContext.class);
context.setApplicationId(appId);
context.setApplicationName(APP_NAME + "-" + appNameSuffix);
if (StringUtils.isBlank(this.queueName)) {
context.setQueue(this.conf.get(DEFAULT_QUEUE_CONFIG,
YarnConfiguration.DEFAULT_QUEUE_NAME));
} else {
context.setQueue(this.queueName);
}
ContainerLaunchContext amContainer =
this.recordFactory.newRecordInstance(ContainerLaunchContext.class);
Resource resource = BuilderUtils.newResource(1024, 1);
context.setResource(resource);
context.setAMContainerSpec(amContainer);
submitRequest.setApplicationSubmissionContext(context);
context.setUnmanagedAM(true);
context.setKeepContainersAcrossApplicationAttempts(
this.keepContainersAcrossApplicationAttempts);
LOG.info("Submitting unmanaged application {}", appId);
this.rmClient.submitApplication(submitRequest);
}
{code}
Finnally, I fixed this by set application's priority to the
ApplicationSubmissionContext. The priority which from the homeCluster's
response when register appMaster to homeCluster's RM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7365) AppMaster register UAM can lost application priority in subCluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yao Guangdong resolved MAPREDUCE-7365.
--
Resolution: Abandoned
> AppMaster register UAM can lost application priority in subCluster
> ---
>
> Key: MAPREDUCE-7365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7365
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, nodemanager, resourcemanager, yarn
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 3.3.0
>Reporter: Yao Guangdong
>Priority: Major
> Fix For: 3.3.2
>
>
> AppMaster register uam to subCluster can lost application priority in yarn
> federation cluster.Which make the subCluster's RM allocate resouce to
> application use default priority regardless of the application's real
> priority.
> By analyzing the code, I found that the appMaster submitApplication to
> subCluster did't set the priority to ApplicationSubmissionContext. As follows:
>
> {code:java}
> private void submitUnmanagedApp(ApplicationId appId)
> throws YarnException, IOException {
> SubmitApplicationRequest submitRequest =
> this.recordFactory.newRecordInstance(SubmitApplicationRequest.class);
> ApplicationSubmissionContext context = this.recordFactory
> .newRecordInstance(ApplicationSubmissionContext.class);
> context.setApplicationId(appId);
> context.setApplicationName(APP_NAME + "-" + appNameSuffix);
> if (StringUtils.isBlank(this.queueName)) {
> context.setQueue(this.conf.get(DEFAULT_QUEUE_CONFIG,
> YarnConfiguration.DEFAULT_QUEUE_NAME));
> } else {
> context.setQueue(this.queueName);
> }
> ContainerLaunchContext amContainer =
> this.recordFactory.newRecordInstance(ContainerLaunchContext.class);
> Resource resource = BuilderUtils.newResource(1024, 1);
> context.setResource(resource);
> context.setAMContainerSpec(amContainer);
> submitRequest.setApplicationSubmissionContext(context);
> context.setUnmanagedAM(true);
> context.setKeepContainersAcrossApplicationAttempts(
> this.keepContainersAcrossApplicationAttempts);
> LOG.info("Submitting unmanaged application {}", appId);
> this.rmClient.submitApplication(submitRequest);
> }
> {code}
> Finnally, I fixed this by set application's priority to the
> ApplicationSubmissionContext. The priority which from the homeCluster's
> response when register appMaster to homeCluster's RM.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-compile-javac-root.txt [496K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-mvnsite-root.txt [584K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-patch-pylint.txt [48K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/diff-patch-shelldocs.txt [48K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-javadoc-root.txt [32K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [232K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [424K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [40K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/451/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [28K]
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/ [Oct 13, 2021 4:11:21 PM] (noreply) HDFS-16267. Make hdfs_df tool cross platform (#3542) [Oct 13, 2021 8:44:00 PM] (Konstantin Shvachko) HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) [Oct 14, 2021 1:14:03 AM] (noreply) HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) [Oct 14, 2021 4:07:24 AM] (noreply) HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) [Oct 14, 2021 8:43:32 AM] (noreply) HADOOP-17962. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-tools modules (#3540) [Oct 14, 2021 9:03:01 AM] (noreply) HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules (#3541) [Oct 14, 2021 10:04:00 AM] (noreply) HADOOP-17123. remove guava Preconditions from Hadoop-common-project modules (#3543) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestSnapshotCommands hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS hadoop.hdfs.TestHDFSFileSystemContract hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure hadoop.yarn.csi.client.TestCsiClient hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-compile-javac-root.txt [356K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/results-javadoc-javadoc-root.txt [408K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [580K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [176K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-csi.txt [24K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/658/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [12K]
