This is an automated email from the ASF dual-hosted git repository.
zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new ffb987013 [#1202] improvement: Add HealthScriptChecker for execute
special health check shell script (#1203)
ffb987013 is described below
commit ffb9870131b0702fcbbc364e68edf008542c1481
Author: xumanbu <[email protected]>
AuthorDate: Fri Sep 22 19:34:21 2023 +0800
[#1202] improvement: Add HealthScriptChecker for execute special health
check shell script (#1203)
### What changes were proposed in this pull request?
add HealthScriptChecker for execute special health check shell script
ref:
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/NodeHealthScriptRunner.java
### Why are the changes needed?
Fix: #1202
### Does this PR introduce _any_ user-facing change?
`rss.server.health.checker.script.path` : The health script file path for
HealthScriptChecker, if script file should have execute permission.
`rss.server.health.checker.script.execi.timeout`: The health script file
execute timeout seconds
### How was this patch tested?
add ut
---------
Co-authored-by: jam.xu <[email protected]>
---
docs/server_guide.md | 3 +-
.../apache/uniffle/server/HealthScriptChecker.java | 100 +++++++++++++++++++++
.../apache/uniffle/server/ShuffleServerConf.java | 13 +++
.../uniffle/server/HealthScriptCheckerTest.java | 77 ++++++++++++++++
server/src/test/resources/healthy-script1.sh | 20 +++++
server/src/test/resources/healthy-script2.sh | 20 +++++
server/src/test/resources/healthy-script3.sh | 20 +++++
server/src/test/resources/healthy-script4.sh | 20 +++++
8 files changed, 272 insertions(+), 1 deletion(-)
diff --git a/docs/server_guide.md b/docs/server_guide.md
index 96e6f6c25..f4b5efb82 100644
--- a/docs/server_guide.md
+++ b/docs/server_guide.md
@@ -107,7 +107,8 @@ This document will introduce how to deploy Uniffle shuffle
servers.
|rss.server.storageMediaProvider.from.env.key|-| Sometimes, the local storage
type/media info is provided by external system. RSS would read the env key
defined by this configuration and get info about the storage media of its
basePaths |
|rss.server.decommission.check.interval|60000| The interval(ms) to check if
all applications have finish when server is decommissioning
|
|rss.server.decommission.shutdown|true| Whether shutdown the server after
server is decommissioned
|
-
+|rss.server.health.checker.script.path| - | The health script path for
`HealthScriptChecker`. To enable `HealthScriptChecker`, need to set
`rss.server.health.checker.class.names` and set
`rss.server.health.check.enable` to true.|
+|rss.server.health.checker.script.execute.timeout| 5000 | Timeout for
`HealthScriptChecker` execute health script.(ms)|
### Huge Partition Optimization
A huge partition is a common problem for Spark/MR and so on, caused by data
skew. And it can cause the shuffle server to become unstable. To solve this, we
introduce some mechanisms to limit the writing of huge partitions to avoid
affecting regular partitions, more details can be found in
[ISSUE-378](https://github.com/apache/incubator-uniffle/issues/378). The basic
rules for limiting large partitions are memory usage limits and flushing
individual buffers directly to persistent storage.
diff --git
a/server/src/main/java/org/apache/uniffle/server/HealthScriptChecker.java
b/server/src/main/java/org/apache/uniffle/server/HealthScriptChecker.java
new file mode 100644
index 000000000..c5c58a890
--- /dev/null
+++ b/server/src/main/java/org/apache/uniffle/server/HealthScriptChecker.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.server;
+
+import org.apache.hadoop.util.NodeHealthScriptRunner;
+import org.apache.hadoop.util.Shell;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.uniffle.common.exception.RssException;
+
+public class HealthScriptChecker extends Checker {
+ private static final Logger LOG =
LoggerFactory.getLogger(HealthScriptChecker.class);
+ private String healthScriptPath;
+ private static final String ERROR_PATTERN = "ERROR";
+ private long scriptTimeout;
+
+ HealthScriptChecker(ShuffleServerConf conf) {
+ super(conf);
+ this.healthScriptPath =
conf.getString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH);
+ if (!NodeHealthScriptRunner.shouldRun(healthScriptPath)) {
+ LOG.error(
+ "Rss health check script:"
+ + healthScriptPath
+ + " is not available "
+ + "or doesn't have execute permission, so abort server.");
+ throw new RssException("Health script not available.");
+ }
+ this.scriptTimeout =
conf.getLong(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT);
+ }
+
+ @Override
+ public boolean checkIsHealthy() {
+ HealthCheckerExitStatus status = HealthCheckerExitStatus.SUCCESS;
+ // always build executor is to load the latest script, the script file can
update in need.
+ Shell.ShellCommandExecutor commandExecutor =
+ new Shell.ShellCommandExecutor(new String[] {healthScriptPath}, null,
null, scriptTimeout);
+ try {
+ commandExecutor.execute();
+ } catch (Shell.ExitCodeException e) {
+ // ignore the exit code of the script
+ status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
+ // On Windows, we will not hit the Stream closed IOException
+ // thrown by stdout buffered reader for timeout event.
+ if (Shell.WINDOWS && commandExecutor.isTimedOut()) {
+ status = HealthCheckerExitStatus.TIMED_OUT;
+ }
+ } catch (Exception e) {
+ LOG.warn("execute health script exception, please check script.", e);
+ if (!commandExecutor.isTimedOut()) {
+ status = HealthCheckerExitStatus.FAILED_WITH_EXCEPTION;
+ } else {
+ status = HealthCheckerExitStatus.TIMED_OUT;
+ }
+ } finally {
+ if (status == HealthCheckerExitStatus.SUCCESS) {
+ if (hasErrors(commandExecutor.getOutput())) {
+ status = HealthCheckerExitStatus.FAILED;
+ }
+ }
+ }
+ if (status != HealthCheckerExitStatus.SUCCESS) {
+ LOG.warn("health script check failed. exit status : " + status);
+ }
+ return status == HealthCheckerExitStatus.SUCCESS;
+ }
+
+ private boolean hasErrors(String output) {
+ String[] splits = output.split("\n");
+ for (String split : splits) {
+ if (split.startsWith(ERROR_PATTERN)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private enum HealthCheckerExitStatus {
+ SUCCESS,
+ TIMED_OUT,
+ FAILED_WITH_EXIT_CODE,
+ FAILED_WITH_EXCEPTION,
+ FAILED
+ }
+}
diff --git
a/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
b/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
index 60978ea9a..0a2028478 100644
--- a/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
+++ b/server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java
@@ -271,6 +271,19 @@ public class ShuffleServerConf extends RssBaseConf {
.noDefaultValue()
.withDescription("The list of the Checker's name");
+ public static final ConfigOption<String> HEALTH_CHECKER_SCRIPT_PATH =
+ ConfigOptions.key("rss.server.health.checker.script.path")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ "The health script file path for HealthScriptChecker, if script
file should have execute permission.");
+
+ public static final ConfigOption<Long> HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT
=
+ ConfigOptions.key("rss.server.health.checker.script.execute.timeout")
+ .longType()
+ .defaultValue(5000L)
+ .withDescription("The health script file execute timeout ms.");
+
public static final ConfigOption<Double>
SERVER_MEMORY_SHUFFLE_LOWWATERMARK_PERCENTAGE =
ConfigOptions.key("rss.server.memory.shuffle.lowWaterMark.percentage")
.doubleType()
diff --git
a/server/src/test/java/org/apache/uniffle/server/HealthScriptCheckerTest.java
b/server/src/test/java/org/apache/uniffle/server/HealthScriptCheckerTest.java
new file mode 100644
index 000000000..7c865f7b2
--- /dev/null
+++
b/server/src/test/java/org/apache/uniffle/server/HealthScriptCheckerTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.server;
+
+import java.io.File;
+import java.util.Objects;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class HealthScriptCheckerTest {
+ private String healthScriptTemplate1;
+ private String healthScriptTemplate2;
+ private String healthScriptTemplate3;
+ private String healthScriptTemplate4;
+
+ @BeforeEach
+ void setUp() {
+ healthScriptTemplate1 = getScriptFilePath("healthy-script1.sh");
+ healthScriptTemplate2 = getScriptFilePath("healthy-script2.sh");
+ healthScriptTemplate3 = getScriptFilePath("healthy-script3.sh");
+ healthScriptTemplate4 = getScriptFilePath("healthy-script4.sh");
+ setExecutable(healthScriptTemplate1);
+ setExecutable(healthScriptTemplate2);
+ setExecutable(healthScriptTemplate3);
+ setExecutable(healthScriptTemplate4);
+ }
+
+ @Test
+ void checkIsHealthy() {
+ ShuffleServerConf conf = new ShuffleServerConf();
+ conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH,
healthScriptTemplate1);
+ HealthScriptChecker checker = new HealthScriptChecker(conf);
+ assertTrue(checker.checkIsHealthy());
+
+ conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH,
healthScriptTemplate2);
+ checker = new HealthScriptChecker(conf);
+ assertFalse(checker.checkIsHealthy());
+
+ conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH,
healthScriptTemplate3);
+ checker = new HealthScriptChecker(conf);
+ assertFalse(checker.checkIsHealthy());
+
+ conf.setString(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_PATH,
healthScriptTemplate4);
+ conf.setLong(ShuffleServerConf.HEALTH_CHECKER_SCRIPT_EXECUTE_TIMEOUT,
3000L);
+ checker = new HealthScriptChecker(conf);
+ assertFalse(checker.checkIsHealthy());
+ }
+
+ private String getScriptFilePath(String fileName) {
+ return
Objects.requireNonNull(this.getClass().getClassLoader().getResource(fileName)).getFile();
+ }
+
+ private boolean setExecutable(String filePath) {
+ File file = new File(filePath);
+ file.setExecutable(true);
+ return file.canExecute();
+ }
+}
diff --git a/server/src/test/resources/healthy-script1.sh
b/server/src/test/resources/healthy-script1.sh
new file mode 100644
index 000000000..8adea7ad5
--- /dev/null
+++ b/server/src/test/resources/healthy-script1.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+echo "NORMAL: Check Disk is Normal"
\ No newline at end of file
diff --git a/server/src/test/resources/healthy-script2.sh
b/server/src/test/resources/healthy-script2.sh
new file mode 100644
index 000000000..6648a5804
--- /dev/null
+++ b/server/src/test/resources/healthy-script2.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+echo "ERROR(disk_io): disk with Input/output error"
\ No newline at end of file
diff --git a/server/src/test/resources/healthy-script3.sh
b/server/src/test/resources/healthy-script3.sh
new file mode 100644
index 000000000..75ab0fba5
--- /dev/null
+++ b/server/src/test/resources/healthy-script3.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+exit 1
\ No newline at end of file
diff --git a/server/src/test/resources/healthy-script4.sh
b/server/src/test/resources/healthy-script4.sh
new file mode 100644
index 000000000..c011fed47
--- /dev/null
+++ b/server/src/test/resources/healthy-script4.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+sleep 5s
\ No newline at end of file
