This is an automated email from the ASF dual-hosted git repository.
fanjia pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new f165b182e [Feature][Zeta] Add a jvm.properties file to define the
SeaTunnel Zeta JVM Options (#3771)
f165b182e is described below
commit f165b182ed6bc7c368be9d005e4d0faa7ffd926c
Author: monster <[email protected]>
AuthorDate: Fri Feb 10 14:32:44 2023 +0800
[Feature][Zeta] Add a jvm.properties file to define the SeaTunnel Zeta JVM
Options (#3771)
---
config/jvm_options | 105 +++++++++
.../connector-v2/Error-Quick-Reference-Manual.md | 34 +--
release-note.md | 1 +
.../common/exception/CommonErrorCode.java | 3 +-
.../src/main/bin/seatunnel-cluster.sh | 8 +-
.../seatunnel-starter/src/main/bin/seatunnel.sh | 8 +-
.../core/starter/seatunnel/jvm/JavaVersion.java | 52 +++++
.../starter/seatunnel/jvm/JvmOptionsParser.java | 241 +++++++++++++++++++++
.../seatunnel/jvm/JvmOptionsParserException.java | 36 +++
.../core/starter/seatunnel/jvm/TempDirectory.java | 58 +++++
.../seatunnel/jvm/JvmOptionsParserTests.java | 217 +++++++++++++++++++
.../src/main/resources/jvm_options | 105 +++++++++
12 files changed, 847 insertions(+), 21 deletions(-)
diff --git a/config/jvm_options b/config/jvm_options
new file mode 100644
index 000000000..0194746b4
--- /dev/null
+++ b/config/jvm_options
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+#-Xms4g
+#-Xmx4g
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+#8-13:-XX:+UseConcMarkSweepGC
+#8-13:-XX:CMSInitiatingOccupancyFraction=75
+#8-13:-XX:+UseCMSInitiatingOccupancyOnly
+
+## G1GC Configuration
+# NOTE: G1 GC is only supported on JDK version 10 or later
+# to use G1GC, uncomment the next two lines and update the version on the
+# following three lines to your version of the JDK
+# 10-13:-XX:-UseConcMarkSweepGC
+# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
+#14-:-XX:+UseG1GC
+#14-:-XX:G1ReservePercent=25
+#14-:-XX:InitiatingHeapOccupancyPercent=30
+
+## optimizations
+
+# pre-touch memory pages used by the JVM during initialization
+# -XX:+AlwaysPreTouch
+
+## basic
+
+# explicitly set the stack size
+# -Xss1m
+
+# turn off a JDK optimization that throws away stack traces for common
+# exceptions because stack traces are important for debugging
+# -XX:-OmitStackTraceInFastThrow
+
+# enable helpful NullPointerExceptions (https://openjdk.java.net/jeps/358), if
+# they are supported
+# 14-:-XX:+ShowCodeDetailsInExceptionMessages
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+# -XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps; ensure the directory exists and
+# has sufficient space
+# ${heap.dump.path}
+
+# specify an alternative path for JVM fatal error logs
+# ${error.file}
+
+## JDK 8 GC logging
+
+8:-XX:+PrintGCDetails
+8:-XX:+PrintGCDateStamps
+8:-XX:+PrintTenuringDistribution
+8:-XX:+PrintGCApplicationStoppedTime
+8:-Xloggc:${loggc}
+8:-XX:+UseGCLogFileRotation
+8:-XX:NumberOfGCLogFiles=32
+8:-XX:GCLogFileSize=64m
\ No newline at end of file
diff --git a/docs/en/connector-v2/Error-Quick-Reference-Manual.md
b/docs/en/connector-v2/Error-Quick-Reference-Manual.md
index ad9e5acec..1e4932937 100644
--- a/docs/en/connector-v2/Error-Quick-Reference-Manual.md
+++ b/docs/en/connector-v2/Error-Quick-Reference-Manual.md
@@ -16,22 +16,24 @@ problems encountered by users.
## SeaTunnel Common Error Codes
-| code | description
| solution
|
-|-----------|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| COMMON-01 | File operation failed, such as (read,list,write,move,copy,sync)
etc... | When users encounter this error code, it is usually there are some
problems in the file operation, please check if the file is OK
|
-| COMMON-02 | Json covert/parse operation failed
| When users encounter this error code, it is usually there are some
problems about json converting or parsing, please check if the json format is
correct |
-| COMMON-03 | Reflect class operation failed
| When users encounter this error code, it is usually there are some
problems on class reflect operation, please check the jar dependency whether
exists in classpath |
-| COMMON-04 | Serialize class operation failed
| When users encounter this error code, it is usually there are some
problems on class serialize operation, please check java environment
|
-| COMMON-05 | Unsupported operation
| When users encounter this error code, users may trigger an unsupported
operation such as enabled some unsupported features
|
-| COMMON-06 | Illegal argument
| When users encounter this error code, it maybe user-configured
parameters are not legal, please correct it according to the tips
|
-| COMMON-07 | Unsupported data type
| When users encounter this error code, it maybe connectors don't support
this data type
|
-| COMMON-08 | Sql operation failed, such as (execute,addBatch,close) etc...
| When users encounter this error code, it is usually there are some
problems on sql execute process, please check the sql whether correct
|
-| COMMON-09 | Get table schema from upstream data failed
| When users encounter this error code, it maybe SeaTunnel try to get
schema information from connector source data failed, please check your
configuration whether correct and connector is work |
-| COMMON-10 | Flush data operation that in sink connector failed
| When users encounter this error code, it maybe SeaTunnel try to flush
batch data to sink connector field, please check your configuration whether
correct and connector is work |
-| COMMON-11 | Sink writer operation failed, such as (open, close) etc...
| When users encounter this error code, it maybe some operation of writer
such as Parquet,Orc,IceBerg failed, you need to check if the corresponding file
or resource has read and write permissions |
-| COMMON-12 | Source reader operation failed, such as (open, close) etc...
| When users encounter this error code, it maybe some operation of reader
such as Parquet,Orc,IceBerg failed, you need to check if the corresponding file
or resource has read and write permissions |
-| COMMON-13 | Http operation failed, such as (open, close, response) etc...
| When users encounter this error code, it maybe some http requests
failed, please check your network environment
|
-| COMMON-14 | Class load operation failed
| When users encounter this error code, it maybe some The corresponding
jar does not exist, or the type is not supported
|
+| code | description
| solution
|
+|-----------|-------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| COMMON-01 | File operation failed, such as (read,list,write,move,copy,sync)
etc... | When users encounter this error code, it is usually there are some
problems in the file operation, please check if the file is OK
|
+| COMMON-02 | Json covert/parse operation failed
| When users encounter this error code, it is usually there are some
problems about json converting or parsing, please check if the json format is
correct |
+| COMMON-03 | Reflect class operation failed
| When users encounter this error code, it is usually there are some
problems on class reflect operation, please check the jar dependency whether
exists in classpath |
+| COMMON-04 | Serialize class operation failed
| When users encounter this error code, it is usually there are some
problems on class serialize operation, please check java environment
|
+| COMMON-05 | Unsupported operation
| When users encounter this error code, users may trigger an unsupported
operation such as enabled some unsupported features
|
+| COMMON-06 | Illegal argument
| When users encounter this error code, it maybe user-configured
parameters are not legal, please correct it according to the tips
|
+| COMMON-07 | Unsupported data type
| When users encounter this error code, it maybe connectors don't
support this data type
|
+| COMMON-08 | Sql operation failed, such as (execute,addBatch,close) etc...
| When users encounter this error code, it is usually there are some
problems on sql execute process, please check the sql whether correct
|
+| COMMON-09 | Get table schema from upstream data failed
| When users encounter this error code, it maybe SeaTunnel try to get
schema information from connector source data failed, please check your
configuration whether correct and connector is work |
+| COMMON-10 | Flush data operation that in sink connector failed
| When users encounter this error code, it maybe SeaTunnel try to flush
batch data to sink connector field, please check your configuration whether
correct and connector is work |
+| COMMON-11 | Sink writer operation failed, such as (open, close) etc...
| When users encounter this error code, it maybe some operation of
writer such as Parquet,Orc,IceBerg failed, you need to check if the
corresponding file or resource has read and write permissions |
+| COMMON-12 | Source reader operation failed, such as (open, close) etc...
| When users encounter this error code, it maybe some operation of
reader such as Parquet,Orc,IceBerg failed, you need to check if the
corresponding file or resource has read and write permissions |
+| COMMON-13 | Http operation failed, such as (open, close, response) etc...
| When users encounter this error code, it maybe some http requests
failed, please check your network environment
|
+| COMMON-14 | Kerberos authorized failed
| When users encounter this error code, it maybe some The Kerberos
authorized is misconfigured
|
+| COMMON-15 | Class load operation failed
| When users encounter this error code, it maybe some The corresponding
jar does not exist, or the type is not supported
|
+| COMMON-16 | Encountered improperly formatted JVM option
| When users encounter this error code, it maybe some The JVM option
formatted improperly
|
## Assert Connector Error Codes
diff --git a/release-note.md b/release-note.md
index c9c3eace9..d0a337566 100644
--- a/release-note.md
+++ b/release-note.md
@@ -4,6 +4,7 @@
### Zeta Engine
- [Script]Add support close engine instance shell
- [Client]Add Zeta Client ShutdownHook To Cancel Job
+- [Script]Add a jvm.properties file to define the SeaTunnel Zeta JVM Options
### Core
- [Starter][Flink]Support transform-v2 for flink #3396
- [Flink] Support flink 1.14.x #3963
diff --git
a/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java
b/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java
index 9fdad8139..8b178996b 100644
---
a/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java
+++
b/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonErrorCode.java
@@ -32,7 +32,8 @@ public enum CommonErrorCode implements SeaTunnelErrorCode {
READER_OPERATION_FAILED("COMMON-12", "Source reader operation failed, such
as (open, close) etc..."),
HTTP_OPERATION_FAILED("COMMON-13", "Http operation failed, such as (open,
close, response) etc..."),
KERBEROS_AUTHORIZED_FAILED("COMMON-14", "Kerberos authorized failed"),
- CLASS_NOT_FOUND("COMMON-15", "Class load operation failed");
+ CLASS_NOT_FOUND("COMMON-15", "Class load operation failed"),
+ IMPROPERLY_FORMATTED_JVM_OPTION("COMMON-16", "Encountered improperly
formatted JVM option");
private final String code;
private final String description;
diff --git a/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh
b/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh
index eaeb2c03e..f8945677f 100755
--- a/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh
+++ b/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel-cluster.sh
@@ -84,8 +84,12 @@ if [ -e "${CONF_DIR}/log4j2.properties" ]; then
JAVA_OPTS="${JAVA_OPTS} -Dseatunnel.logs.file_name=seatunnel-engine-server"
fi
-echo "JAVA_OPTS: ${JAVA_OPTS}"
-
CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR}
+ST_TMPDIR=`java -cp ${CLASS_PATH}
org.apache.seatunnel.core.starter.seatunnel.jvm.TempDirectory`
+# The JVM options parser produces the final JVM options to start
seatunnel-engine.
+JVM_OPTIONS=`java -cp ${CLASS_PATH}
org.apache.seatunnel.core.starter.seatunnel.jvm.JvmOptionsParser ${CONF_DIR}`
+JAVA_OPTS="${JAVA_OPTS} ${JVM_OPTIONS//\$\{loggc\}/${ST_TMPDIR}}"
+echo "JAVA_OPTS:" ${JAVA_OPTS}
+
java ${JAVA_OPTS} -cp ${CLASS_PATH} ${APP_MAIN} ${args}
diff --git a/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel.sh
b/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel.sh
index ee79c725d..d5b50250c 100755
--- a/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel.sh
+++ b/seatunnel-core/seatunnel-starter/src/main/bin/seatunnel.sh
@@ -92,8 +92,12 @@ if [ -e "${CONF_DIR}/log4j2_client.properties" ]; then
fi
fi
-echo "JAVA_OPTS: ${JAVA_OPTS}"
-
CLASS_PATH=${APP_DIR}/lib/*:${APP_JAR}
+ST_TMPDIR=`java -cp ${CLASS_PATH}
org.apache.seatunnel.core.starter.seatunnel.jvm.TempDirectory`
+# The JVM options parser produces the final JVM options to start
seatunnel-engine.
+JVM_OPTIONS=`java -cp ${CLASS_PATH}
org.apache.seatunnel.core.starter.seatunnel.jvm.JvmOptionsParser ${CONF_DIR}`
+JAVA_OPTS="${JAVA_OPTS} ${JVM_OPTIONS//\$\{loggc\}/${ST_TMPDIR}}"
+echo "JAVA_OPTS:" ${JAVA_OPTS}
+
java ${JAVA_OPTS} -cp ${CLASS_PATH} ${APP_MAIN} ${args}
diff --git
a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JavaVersion.java
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JavaVersion.java
new file mode 100644
index 000000000..62c962e0b
--- /dev/null
+++
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JavaVersion.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.core.starter.seatunnel.jvm;
+
+import org.apache.seatunnel.common.exception.CommonErrorCode;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+public class JavaVersion {
+
+ public static final List<Integer> CURRENT =
parse(System.getProperty("java.specification.version"));
+
+ static List<Integer> parse(final String value) {
+ if (!value.matches("^0*[0-9]+(\\.[0-9]+)*$")) {
+ throw new
JvmOptionsParserException(CommonErrorCode.ILLEGAL_ARGUMENT, String.format("JAVA
version [%s] of the system is incorrect", value));
+ }
+
+ final List<Integer> version = new ArrayList<Integer>();
+ final String[] components = value.split("\\.");
+ for (final String component : components) {
+ version.add(Integer.valueOf(component));
+ }
+ return version;
+ }
+
+ public static int majorVersion(final List<Integer> javaVersion) {
+ Objects.requireNonNull(javaVersion);
+ if (javaVersion.get(0) > 1) {
+ return javaVersion.get(0);
+ } else {
+ return javaVersion.get(1);
+ }
+ }
+
+}
diff --git
a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParser.java
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParser.java
new file mode 100644
index 000000000..e9b236675
--- /dev/null
+++
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParser.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.core.starter.seatunnel.jvm;
+
+import org.apache.seatunnel.common.exception.CommonErrorCode;
+
+import lombok.SneakyThrows;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parses JVM options from a file and prints a single line with all JVM
options to standard output.
+ */
+@SuppressWarnings("checkstyle:InnerTypeLast")
+final class JvmOptionsParser {
+
+ /**
+ * The main entry point. The exit code is 0 if the JVM options were
successfully parsed, otherwise the exit code is 1. If an improperly
+ * formatted line is discovered, the line is output to standard error.
+ */
+ @SuppressWarnings("checkstyle:RegexpSingleline")
+ public static void main(final String[] args) {
+ if (args.length != 1) {
+ throw new IllegalArgumentException(
+ "Expected one arguments specifying path to PATH_CONF, but was
" + Arrays.toString(args)
+ );
+ }
+
+ final JvmOptionsParser parser = new JvmOptionsParser();
+ final List<String> jvmOptions =
parser.readJvmOptionsFiles(Paths.get(args[0]));
+ System.out.println(String.join(" ", jvmOptions));
+ }
+
+ @SneakyThrows
+ List<String> readJvmOptionsFiles(final Path config) {
+ final ArrayList<Path> jvmOptionsFiles = new ArrayList<>();
+ jvmOptionsFiles.add(config.resolve("jvm_options"));
+
+ final List<String> jvmOptions = new ArrayList<>();
+
+ for (final Path jvmOptionsFile : jvmOptionsFiles) {
+ final SortedMap<Integer, String> invalidLines = new TreeMap<>();
+ try (
+ InputStream is = Files.newInputStream(jvmOptionsFile);
+ Reader reader = new InputStreamReader(is,
StandardCharsets.UTF_8);
+ BufferedReader br = new BufferedReader(reader)
+ ) {
+ parse(JavaVersion.majorVersion(JavaVersion.CURRENT), br,
jvmOptions::add, invalidLines::put);
+ }
+ if (!invalidLines.isEmpty()) {
+ throw new
JvmOptionsParserException(CommonErrorCode.IMPROPERLY_FORMATTED_JVM_OPTION,
String.format(
+ Locale.ROOT,
+ "encountered [%d] error%s parsing [%s]",
+ invalidLines.size(),
+ invalidLines.size() == 1 ? "" : "s",
+ jvmOptionsFile));
+ }
+ }
+ return jvmOptions;
+ }
+
+ /**
+ * Callback for valid JVM options.
+ */
+ interface JvmOptionConsumer {
+ /**
+ * Invoked when a line in the JVM options file matches the specified
syntax and the specified major version.
+ * @param jvmOption the matching JVM option
+ */
+ void accept(String jvmOption);
+ }
+
+ /**
+ * Callback for invalid lines in the JVM options.
+ */
+ interface InvalidLineConsumer {
+ /**
+ * Invoked when a line in the JVM options does not match the specified
syntax.
+ */
+ void accept(int lineNumber, String line);
+ }
+
+ private static final Pattern PATTERN =
Pattern.compile("((?<start>\\d+)(?<range>-)?(?<end>\\d+)?:)?(?<option>-.*)$");
+
+ /**
+ * Parse the line-delimited JVM options from the specified buffered reader
for the specified Java major version.
+ * Valid JVM options are:
+ * <ul>
+ * <li>
+ * a line starting with a dash is treated as a JVM option that
applies to all versions
+ * </li>
+ * <li>
+ * a line starting with a number followed by a colon is treated as
a JVM option that applies to the matching Java major version
+ * only
+ * </li>
+ * <li>
+ * a line starting with a number followed by a dash followed by a
colon is treated as a JVM option that applies to the matching
+ * Java specified major version and all larger Java major versions
+ * </li>
+ * <li>
+ * a line starting with a number followed by a dash followed by a
number followed by a colon is treated as a JVM option that
+ * applies to the specified range of matching Java major versions
+ * </li>
+ * </ul>
+ *
+ * For example, if the specified Java major version is 8, the following
JVM options will be accepted:
+ * <ul>
+ * <li>
+ * {@code -XX:+PrintGCDateStamps}
+ * </li>
+ * <li>
+ * {@code 8:-XX:+PrintGCDateStamps}
+ * </li>
+ * <li>
+ * {@code 8-:-XX:+PrintGCDateStamps}
+ * </li>
+ * <li>
+ * {@code 7-8:-XX:+PrintGCDateStamps}
+ * </li>
+ * </ul>
+ * and the following JVM options will not be accepted:
+ * <ul>
+ * <li>
+ * {@code
9:-Xlog:age*=trace,gc*,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m}
+ * </li>
+ * <li>
+ * {@code
9-:-Xlog:age*=trace,gc*,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m}
+ * </li>
+ * <li>
+ * {@code
9-10:-Xlog:age*=trace,gc*,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m}
+ * </li>
+ * </ul>
+ *
+ * If the version syntax specified on a line matches the specified JVM
options, the JVM option callback will be invoked with the JVM
+ * option. If the line does not match the specified syntax for the JVM
options, the invalid line callback will be invoked with the
+ * contents of the entire line.
+ *
+ * @param javaMajorVersion the Java major version to match JVM options
against
+ * @param br the buffered reader to read line-delimited JVM options from
+ * @param jvmOptionConsumer the callback that accepts matching JVM options
+ * @param invalidLineConsumer a callback that accepts invalid JVM options
+ * @throws IOException if an I/O exception occurs reading from the
buffered reader
+ */
+ @SneakyThrows
+ static void parse(
+ final int javaMajorVersion,
+ final BufferedReader br,
+ final JvmOptionConsumer jvmOptionConsumer,
+ final InvalidLineConsumer invalidLineConsumer) {
+ int lineNumber = 0;
+ while (true) {
+ final String line = br.readLine();
+ lineNumber++;
+ if (line == null) {
+ break;
+ }
+ if (line.startsWith("#")) {
+ // lines beginning with "#" are treated as comments
+ continue;
+ }
+ if (line.matches("\\s*")) {
+ // skip blank lines
+ continue;
+ }
+ final Matcher matcher = PATTERN.matcher(line);
+ if (matcher.matches()) {
+ final String start = matcher.group("start");
+ final String end = matcher.group("end");
+ if (start == null) {
+ // no range present, unconditionally apply the JVM option
+ jvmOptionConsumer.accept(line);
+ } else {
+ final int lower;
+ try {
+ lower = Integer.parseInt(start);
+ } catch (final NumberFormatException e) {
+ invalidLineConsumer.accept(lineNumber, line);
+ continue;
+ }
+ final int upper;
+ if (matcher.group("range") == null) {
+ // no range is present, apply the JVM option to the
specified major version only
+ upper = lower;
+ } else if (end == null) {
+ // a range of the form \\d+- is present, apply the JVM
option to all major versions larger than the specified one
+ upper = Integer.MAX_VALUE;
+ } else {
+ // a range of the form \\d+-\\d+ is present, apply the
JVM option to the specified range of major versions
+ try {
+ upper = Integer.parseInt(end);
+ } catch (final NumberFormatException e) {
+ invalidLineConsumer.accept(lineNumber, line);
+ continue;
+ }
+ if (upper < lower) {
+ invalidLineConsumer.accept(lineNumber, line);
+ continue;
+ }
+ }
+ if (lower <= javaMajorVersion && javaMajorVersion <=
upper) {
+ jvmOptionConsumer.accept(matcher.group("option"));
+ }
+ }
+ } else {
+ invalidLineConsumer.accept(lineNumber, line);
+ }
+ }
+ }
+
+}
diff --git
a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserException.java
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserException.java
new file mode 100644
index 000000000..d6ecf235b
--- /dev/null
+++
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserException.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.core.starter.seatunnel.jvm;
+
+import org.apache.seatunnel.common.exception.SeaTunnelErrorCode;
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+
+public class JvmOptionsParserException extends SeaTunnelRuntimeException {
+
+ public JvmOptionsParserException(SeaTunnelErrorCode seaTunnelErrorCode,
String errorMessage) {
+ super(seaTunnelErrorCode, errorMessage);
+ }
+
+ public JvmOptionsParserException(SeaTunnelErrorCode seaTunnelErrorCode,
String errorMessage, Throwable cause) {
+ super(seaTunnelErrorCode, errorMessage, cause);
+ }
+
+ public JvmOptionsParserException(SeaTunnelErrorCode seaTunnelErrorCode,
Throwable cause) {
+ super(seaTunnelErrorCode, cause);
+ }
+}
diff --git
a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/TempDirectory.java
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/TempDirectory.java
new file mode 100644
index 000000000..becff375f
--- /dev/null
+++
b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/jvm/TempDirectory.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.core.starter.seatunnel.jvm;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
+
+/**
+ * Provides a path for a temporary directory. On non-Windows OS, this will be
created as a sub-directory of the default temporary directory.
+ * Note that this causes the created temporary directory to be a private
temporary directory.
+ */
+final class TempDirectory {
+
+ /**
+ * The main entry point. The exit code is 0 if we successfully created a
temporary directory as a sub-directory of the default
+ * temporary directory and printed the resulting path to the console.
+ *
+ * @param args the args to the program which should be empty
+ * @throws IOException if an I/O exception occurred while creating the
temporary directory
+ */
+ @SuppressWarnings("checkstyle:RegexpSingleline")
+ public static void main(final String[] args) throws IOException {
+ if (args.length != 0) {
+ throw new IllegalArgumentException("expected zero arguments but
was " + Arrays.toString(args));
+ }
+ /*
+ * On Windows, we avoid creating a unique temporary directory per
invocation lest we pollute the temporary directory. On other
+ * operating systems, temporary directories will be cleaned
automatically via various mechanisms (e.g., systemd, or restarts).
+ */
+ final Path path;
+ if (System.getProperty("os.name").startsWith("Windows")) {
+ path = Paths.get(System.getProperty("java.io.tmpdir"),
"seatunnel");
+ Files.createDirectories(path);
+ } else {
+ path = Files.createTempDirectory("seatunnel-");
+ }
+ System.out.println(path);
+ }
+
+}
diff --git
a/seatunnel-core/seatunnel-starter/src/test/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserTests.java
b/seatunnel-core/seatunnel-starter/src/test/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserTests.java
new file mode 100644
index 000000000..d51c5e38a
--- /dev/null
+++
b/seatunnel-core/seatunnel-starter/src/test/java/org/apache/seatunnel/core/starter/seatunnel/jvm/JvmOptionsParserTests.java
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.core.starter.seatunnel.jvm;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+@SuppressWarnings("checkstyle:MagicNumber")
+public class JvmOptionsParserTests {
+
+ @Test
+ public void testUnversionedOptions() throws IOException {
+ try (StringReader sr = new StringReader("-Xms1g\n-Xmx1g");
+ BufferedReader br = new BufferedReader(sr)) {
+ assertExpectedJvmOptions(randomIntBetween(8, Integer.MAX_VALUE),
br, Arrays.asList("-Xms1g", "-Xmx1g"));
+ }
+
+ }
+
+ @Test
+ public void testSingleVersionOption() throws IOException {
+ final int javaMajorVersion = randomIntBetween(8, Integer.MAX_VALUE -
1);
+ final int smallerJavaMajorVersion = randomIntBetween(7,
javaMajorVersion);
+ final int largerJavaMajorVersion = randomIntBetween(javaMajorVersion +
1, Integer.MAX_VALUE);
+ try (StringReader sr = new StringReader(
+ String.format(
+ Locale.ROOT,
+ "-Xms1g\n%d:-Xmx1g\n%d:-XX:+UseG1GC\n%d:-Xlog:gc",
+ javaMajorVersion,
+ smallerJavaMajorVersion,
+ largerJavaMajorVersion));
+ BufferedReader br = new BufferedReader(sr)) {
+ assertExpectedJvmOptions(javaMajorVersion, br,
Arrays.asList("-Xms1g", "-Xmx1g"));
+ }
+ }
+
+ @Test
+ public void testUnboundedVersionOption() throws IOException {
+ final int javaMajorVersion = randomIntBetween(8, Integer.MAX_VALUE -
1);
+ final int smallerJavaMajorVersion = randomIntBetween(7,
javaMajorVersion);
+ final int largerJavaMajorVersion = randomIntBetween(javaMajorVersion +
1, Integer.MAX_VALUE);
+ try (StringReader sr = new StringReader(
+ String.format(
+ Locale.ROOT,
+ "-Xms1g\n%d-:-Xmx1g\n%d-:-XX:+UseG1GC\n%d-:-Xlog:gc",
+ javaMajorVersion,
+ smallerJavaMajorVersion,
+ largerJavaMajorVersion));
+ BufferedReader br = new BufferedReader(sr)) {
+ assertExpectedJvmOptions(javaMajorVersion, br,
Arrays.asList("-Xms1g", "-Xmx1g", "-XX:+UseG1GC"));
+ }
+ }
+
+ @Test
+ public void testBoundedVersionOption() throws IOException {
+ final int javaMajorVersion = randomIntBetween(8, Integer.MAX_VALUE -
1);
+ final int javaMajorVersionUpperBound =
randomIntBetween(javaMajorVersion, Integer.MAX_VALUE - 1);
+ final int smallerJavaMajorVersionLowerBound = randomIntBetween(7,
javaMajorVersion);
+ final int smallerJavaMajorVersionUpperBound =
randomIntBetween(smallerJavaMajorVersionLowerBound, javaMajorVersion);
+ final int largerJavaMajorVersionLowerBound =
randomIntBetween(javaMajorVersion + 1, Integer.MAX_VALUE);
+ final int largerJavaMajorVersionUpperBound =
randomIntBetween(largerJavaMajorVersionLowerBound, Integer.MAX_VALUE);
+ try (StringReader sr = new StringReader(
+ String.format(
+ Locale.ROOT,
+
"-Xms1g\n%d-%d:-Xmx1g\n%d-%d:-XX:+UseG1GC\n%d-%d:-Xlog:gc",
+ javaMajorVersion,
+ javaMajorVersionUpperBound,
+ smallerJavaMajorVersionLowerBound,
+ smallerJavaMajorVersionUpperBound,
+ largerJavaMajorVersionLowerBound,
+ largerJavaMajorVersionUpperBound));
+ BufferedReader br = new BufferedReader(sr)) {
+ assertExpectedJvmOptions(javaMajorVersion, br,
Arrays.asList("-Xms1g", "-Xmx1g"));
+ }
+ }
+
+ @Test
+ public void testComplexOptions() throws IOException {
+ final int javaMajorVersion = randomIntBetween(8, Integer.MAX_VALUE -
1);
+ final int javaMajorVersionUpperBound =
randomIntBetween(javaMajorVersion, Integer.MAX_VALUE - 1);
+ final int smallerJavaMajorVersionLowerBound = randomIntBetween(7,
javaMajorVersion);
+ final int smallerJavaMajorVersionUpperBound =
randomIntBetween(smallerJavaMajorVersionLowerBound, javaMajorVersion);
+ final int largerJavaMajorVersionLowerBound =
randomIntBetween(javaMajorVersion + 1, Integer.MAX_VALUE);
+ final int largerJavaMajorVersionUpperBound =
randomIntBetween(largerJavaMajorVersionLowerBound, Integer.MAX_VALUE);
+ try (StringReader sr = new StringReader(
+ String.format(
+ Locale.ROOT,
+
"-Xms1g\n%d:-Xmx1g\n%d-:-XX:+UseG1GC\n%d-%d:-Xlog:gc\n%d-%d:-XX:+PrintFlagsFinal\n%d-%d:-XX+AggressiveOpts",
+ javaMajorVersion,
+ javaMajorVersion,
+ javaMajorVersion,
+ javaMajorVersionUpperBound,
+ smallerJavaMajorVersionLowerBound,
+ smallerJavaMajorVersionUpperBound,
+ largerJavaMajorVersionLowerBound,
+ largerJavaMajorVersionUpperBound));
+ BufferedReader br = new BufferedReader(sr)) {
+ assertExpectedJvmOptions(javaMajorVersion, br,
Arrays.asList("-Xms1g", "-Xmx1g", "-XX:+UseG1GC", "-Xlog:gc"));
+ }
+ }
+
+ @Test
+ private void assertExpectedJvmOptions(final int javaMajorVersion, final
BufferedReader br, final List<String> expectedJvmOptions) {
+ final Map<String, AtomicBoolean> seenJvmOptions = new HashMap<>();
+ for (final String expectedJvmOption : expectedJvmOptions) {
+ Assertions.assertNull(seenJvmOptions.put(expectedJvmOption, new
AtomicBoolean()));
+ }
+ JvmOptionsParser.parse(javaMajorVersion, br, jvmOption -> {
+ final AtomicBoolean seen = seenJvmOptions.get(jvmOption);
+ if (seen == null) {
+ Assertions.fail("unexpected JVM option [" + jvmOption + "]");
+ }
+ Assertions.assertFalse(seen.get(), "saw JVM option [" + jvmOption
+ "] more than once");
+ seen.set(true);
+ }, (lineNumber, line) -> Assertions.fail("unexpected invalid line [" +
line + "] on line number [" + lineNumber + "]"));
+ for (final Map.Entry<String, AtomicBoolean> seenJvmOption :
seenJvmOptions.entrySet()) {
+ Assertions.assertTrue(seenJvmOption.getValue().get(), "expected
JVM option [" + seenJvmOption.getKey() + "]");
+ }
+ }
+
+ @Test
+ public void testInvalidLines() throws IOException {
+ try (StringReader sr = new StringReader("XX:+UseG1GC");
+ BufferedReader br = new BufferedReader(sr)) {
+ JvmOptionsParser.parse(
+ randomIntBetween(8, Integer.MAX_VALUE),
+ br,
+ jvmOption -> Assertions.fail("unexpected valid JVM option [" +
jvmOption + "]"), (lineNumber, line) -> {
+ Assertions.assertEquals(lineNumber, 1);
+ Assertions.assertEquals(line, "XX:+UseG1GC");
+ });
+ }
+ final int javaMajorVersion = randomIntBetween(8, Integer.MAX_VALUE);
+ final int smallerJavaMajorVersion = randomIntBetween(7,
javaMajorVersion - 1);
+ final String invalidRangeLine = String.format(Locale.ROOT,
"%d:%d-XX:+UseG1GC", javaMajorVersion, smallerJavaMajorVersion);
+ try (StringReader sr = new StringReader(invalidRangeLine);
+ BufferedReader br = new BufferedReader(sr)) {
+ assertInvalidLines(br, Collections.singletonMap(1,
invalidRangeLine));
+ }
+
+ final long invalidLowerJavaMajorVersion = (long) randomIntBetween(1,
16) + Integer.MAX_VALUE;
+ final long invalidUpperJavaMajorVersion = (long) randomIntBetween(1,
16) + Integer.MAX_VALUE;
+ final String numberFormatExceptionsLine = String.format(
+ Locale.ROOT,
+ "%d:-XX:+UseG1GC\n8-%d:-XX:+AggressiveOpts",
+ invalidLowerJavaMajorVersion,
+ invalidUpperJavaMajorVersion);
+ try (StringReader sr = new StringReader(numberFormatExceptionsLine);
+ BufferedReader br = new BufferedReader(sr)) {
+ final Map<Integer, String> invalidLines = new HashMap<>(2);
+ invalidLines.put(1, String.format(Locale.ROOT, "%d:-XX:+UseG1GC",
invalidLowerJavaMajorVersion));
+ invalidLines.put(2, String.format(Locale.ROOT,
"8-%d:-XX:+AggressiveOpts", invalidUpperJavaMajorVersion));
+ assertInvalidLines(br, invalidLines);
+ }
+
+ final String multipleInvalidLines = "XX:+UseG1GC\nXX:+AggressiveOpts";
+ try (StringReader sr = new StringReader(multipleInvalidLines);
+ BufferedReader br = new BufferedReader(sr)) {
+ final Map<Integer, String> invalidLines = new HashMap<>(2);
+ invalidLines.put(1, "XX:+UseG1GC");
+ invalidLines.put(2, "XX:+AggressiveOpts");
+ assertInvalidLines(br, invalidLines);
+ }
+
+ final int lowerBound = randomIntBetween(9, 16);
+ final int upperBound = randomIntBetween(8, lowerBound - 1);
+ final String upperBoundGreaterThanLowerBound =
String.format(Locale.ROOT, "%d-%d-XX:+UseG1GC", lowerBound, upperBound);
+ try (StringReader sr = new
StringReader(upperBoundGreaterThanLowerBound);
+ BufferedReader br = new BufferedReader(sr)) {
+ assertInvalidLines(br, Collections.singletonMap(1,
upperBoundGreaterThanLowerBound));
+ }
+ }
+
+ private void assertInvalidLines(final BufferedReader br, final
Map<Integer, String> invalidLines) throws IOException {
+ final Map<Integer, String> seenInvalidLines = new
HashMap<>(invalidLines.size());
+ JvmOptionsParser.parse(
+ randomIntBetween(8, Integer.MAX_VALUE),
+ br,
+ jvmOption -> Assertions.fail("unexpected valid JVM options [" +
jvmOption + "]"),
+ (lineNumber, line) -> seenInvalidLines.put(lineNumber, line));
+ Assertions.assertEquals(seenInvalidLines, invalidLines);
+ }
+
+ /**
+ * Get a random integer between start and end
+ */
+ public static int randomIntBetween(int start, int end) {
+ return (int) (Math.random() * (end - start + 1) + start);
+ }
+
+}
diff --git
a/seatunnel-engine/seatunnel-engine-common/src/main/resources/jvm_options
b/seatunnel-engine/seatunnel-engine-common/src/main/resources/jvm_options
new file mode 100644
index 000000000..d5226384d
--- /dev/null
+++ b/seatunnel-engine/seatunnel-engine-common/src/main/resources/jvm_options
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+# -Xms4g
+# -Xmx4g
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+# 8-13:-XX:+UseConcMarkSweepGC
+# 8-13:-XX:CMSInitiatingOccupancyFraction=75
+# 8-13:-XX:+UseCMSInitiatingOccupancyOnly
+
+## G1GC Configuration
+# NOTE: G1 GC is only supported on JDK version 10 or later
+# to use G1GC, uncomment the next two lines and update the version on the
+# following three lines to your version of the JDK
+# 10-13:-XX:-UseConcMarkSweepGC
+# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
+# 14-:-XX:+UseG1GC
+# 14-:-XX:G1ReservePercent=25
+# 14-:-XX:InitiatingHeapOccupancyPercent=30
+
+## optimizations
+
+# pre-touch memory pages used by the JVM during initialization
+# -XX:+AlwaysPreTouch
+
+## basic
+
+# explicitly set the stack size
+# -Xss1m
+
+# turn off a JDK optimization that throws away stack traces for common
+# exceptions because stack traces are important for debugging
+# -XX:-OmitStackTraceInFastThrow
+
+# enable helpful NullPointerExceptions (https://openjdk.java.net/jeps/358), if
+# they are supported
+# 14-:-XX:+ShowCodeDetailsInExceptionMessages
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+# -XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps; ensure the directory exists and
+# has sufficient space
+# ${heap.dump.path}
+
+# specify an alternative path for JVM fatal error logs
+# ${error.file}
+
+## JDK 8 GC logging
+
+8:-XX:+PrintGCDetails
+8:-XX:+PrintGCDateStamps
+8:-XX:+PrintTenuringDistribution
+8:-XX:+PrintGCApplicationStoppedTime
+8:-Xloggc:${loggc}
+8:-XX:+UseGCLogFileRotation
+8:-XX:NumberOfGCLogFiles=32
+8:-XX:GCLogFileSize=64m
\ No newline at end of file
