[
https://issues.apache.org/jira/browse/KYLIN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634113#comment-17634113
]
ASF GitHub Bot commented on KYLIN-5292:
---------------------------------------
Mukvin commented on code in PR #2016:
URL: https://github.com/apache/kylin/pull/2016#discussion_r1022246211
##########
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:
##########
@@ -2818,4 +2818,9 @@ public String getEncryptCipherIvSpec() {
public boolean isEnabledNoAggQuery() {
return
Boolean.parseBoolean(getOptional("kylin.query.enable-no-aggregate-query",
FALSE));
}
+
+ public boolean sparkDeployModeIsYarnCluster() {
Review Comment:
The name of a boolean method would be good at the prefix 'isXXX'.
So this method name could be 'isYarnClusterForSparkDeployMode'.
##########
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/YarnClientUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.utils;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.kylin.shaded.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * @author zhaoliu4
+ * @date 2022/11/13
+ */
+public class YarnClientUtils {
+ private static final Logger logger =
LoggerFactory.getLogger(YarnClientUtils.class);
+
+ public static void killApplication(String jobId) {
+ String applicationId = null;
+ try (YarnClient yarnClient = YarnClient.createYarnClient()) {
+ Configuration yarnConfiguration = new YarnConfiguration();
+ // bug of yarn : https://issues.apache.org/jira/browse/SPARK-15343
+ yarnConfiguration.set("yarn.timeline-service.enabled", "false");
+ yarnClient.init(yarnConfiguration);
+ yarnClient.start();
+
+ Set<String> types = Sets.newHashSet("SPARK");
+ EnumSet<YarnApplicationState> states =
EnumSet.of(YarnApplicationState.NEW, YarnApplicationState.NEW_SAVING,
+ YarnApplicationState.SUBMITTED,
YarnApplicationState.ACCEPTED, YarnApplicationState.RUNNING);
+ List<ApplicationReport> applicationReports =
yarnClient.getApplications(types, states);
+
+ if (CollectionUtils.isEmpty(applicationReports)) {
+ return;
+ }
+
+ for (ApplicationReport report : applicationReports) {
+ if (report.getName().equalsIgnoreCase("job_step_" + jobId +
"-01")) {
Review Comment:
this place may be good as followed.
```
for (...){
if (!...) {
continue;
}
...
break;
}
```
##########
core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java:
##########
@@ -949,7 +951,7 @@ public String findRunningTableSampleJob(String project,
String tableName) {
try {
job = exeMgt.getJob(jobID);
} catch (RuntimeException e) {
- /**
+ /*
Review Comment:
'/**' is correct.
##########
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/YarnClientUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.utils;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.kylin.shaded.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * @author zhaoliu4
+ * @date 2022/11/13
+ */
+public class YarnClientUtils {
+ private static final Logger logger =
LoggerFactory.getLogger(YarnClientUtils.class);
+
+ public static void killApplication(String jobId) {
+ String applicationId = null;
+ try (YarnClient yarnClient = YarnClient.createYarnClient()) {
+ Configuration yarnConfiguration = new YarnConfiguration();
+ // bug of yarn : https://issues.apache.org/jira/browse/SPARK-15343
+ yarnConfiguration.set("yarn.timeline-service.enabled", "false");
+ yarnClient.init(yarnConfiguration);
+ yarnClient.start();
+
+ Set<String> types = Sets.newHashSet("SPARK");
+ EnumSet<YarnApplicationState> states =
EnumSet.of(YarnApplicationState.NEW, YarnApplicationState.NEW_SAVING,
+ YarnApplicationState.SUBMITTED,
YarnApplicationState.ACCEPTED, YarnApplicationState.RUNNING);
Review Comment:
this part could extract to a new file about YarnApplicationState .
##########
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/YarnClientUtils.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.utils;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.client.api.YarnClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.kylin.shaded.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * @author zhaoliu4
+ * @date 2022/11/13
+ */
+public class YarnClientUtils {
+ private static final Logger logger =
LoggerFactory.getLogger(YarnClientUtils.class);
+
+ public static void killApplication(String jobId) {
+ String applicationId = null;
+ try (YarnClient yarnClient = YarnClient.createYarnClient()) {
+ Configuration yarnConfiguration = new YarnConfiguration();
+ // bug of yarn : https://issues.apache.org/jira/browse/SPARK-15343
Review Comment:
this issue affected on spark 2.0.0, how about kylin4 with spark 3.1.1?
> the task that has been submitted to yarn after discard is still executing
> -------------------------------------------------------------------------
>
> Key: KYLIN-5292
> URL: https://issues.apache.org/jira/browse/KYLIN-5292
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v4.0.1
> Reporter: Liu Zhao
> Priority: Major
> Attachments: image-2022-11-11-20-50-31-861.png,
> image-2022-11-11-20-50-54-135.png, image-2022-11-11-20-53-02-878.png,
> image-2022-11-11-20-53-21-382.png, image-2022-11-11-20-53-38-208.png,
> image-2022-11-11-20-53-57-941.png, screenshot-1.png
>
>
> When I discard a task that has been submitted to Yarn, I find that the task
> has not been killed, and the local Sparksubmit subprocess is still alive. The
> following are my environment, phenomena and troubleshooting:
> env: Job and query are in different processes（server.model=job and
> server.model=query）
> phenomena:
> !image-2022-11-11-20-50-31-861.png!
> !image-2022-11-11-20-50-54-135.png!
> !image-2022-11-11-20-53-02-878.png!
> !image-2022-11-11-20-53-21-382.png!
> !image-2022-11-11-20-53-38-208.png!
> !image-2022-11-11-20-53-57-941.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
