[
https://issues.apache.org/jira/browse/FLINK-35898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yiheng tang updated FLINK-35898:
--------------------------------
Description:
We use Yarn to launch Flink jobs and try to add custom parameters to configure
the log output format in the YarnLogConfigUtil.getLog4jCommand method. However,
when the parameter value contains spaces, it causes the startup to fail.
The start command in the Yarn container is as follows, omitting some irrelevant
parameters.
{code:java}
/bin/bash -c "$JAVA_HOME/bin/java
-Dlog.file="/data08/yarn/userlogs/application_1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log"
-Dlog4j.configuration=file.properties
-Dlog4j.configurationFile=file.properties
-Dlog4j2.isThreadContextMapInheritable=true -Dlog.layout.pattern="%d{yyyy-MM-dd
HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO" {code}
We discovered that when the value of log.layout.pattern contains spaces, the
command is incorrectly truncated, causing the content after the space to be
unrecognized, and therefore the Flink job cannot be launched in the Yarn
container.
{code:java}
{HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO"{code}
Therefore, we suggest using single quotes (') instead of double quotes (") to
wrap the values of parameters in the {{getLogBackCommand}} method and
{{getLog4jCommand}} method of the {{YarnLogConfigUtil}} class.
{code:java}
private static String getLogBackCommand(final String logConfigFilePath) {
final boolean hasLogback =
logConfigFilePath.endsWith(CONFIG_FILE_LOGBACK_NAME);
if (!hasLogback) {
return "";
} return new StringBuilder(
"-Dlog.file=\""
+ ApplicationConstants.LOG_DIR_EXPANSION_VAR
+ "/jobmanager.log\"")
.append(" -Dlogback.configurationFile=file:" +
CONFIG_FILE_LOGBACK_NAME)
.toString();
} private static String getLog4jCommand(final String logConfigFilePath) {
final boolean hasLog4j =
logConfigFilePath.endsWith(CONFIG_FILE_LOG4J_NAME);
if (!hasLog4j) {
return "";
} return new StringBuilder(
"-Dlog.file=\""
+ ApplicationConstants.LOG_DIR_EXPANSION_VAR
+ "/jobmanager.log\"")
.append(" -Dlog4j.configuration=file:" + CONFIG_FILE_LOG4J_NAME)
.append(" -Dlog4j.configurationFile=file:" +
CONFIG_FILE_LOG4J_NAME)
.toString();
}
} {code}
---
!image-2024-07-25-21-52-26-973.png|width=843,height=506!!image-2024-07-25-21-51-50-072.png|width=417,height=218!
was:
We use Yarn to launch Flink jobs and try to add custom parameters to configure
the log output format in the YarnLogConfigUtil.getLog4jCommand method. However,
when the parameter value contains spaces, it causes the startup to fail.
The start command in the Yarn container is as follows, omitting some irrelevant
parameters.
{code:java}
/bin/bash -c "$JAVA_HOME/bin/java
-Dlog.file="/data08/yarn/userlogs/application_1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log"
-Dlog4j.configuration=file.properties
-Dlog4j.configurationFile=file.properties
-Dlog4j2.isThreadContextMapInheritable=true -Dlog.layout.pattern="%d{yyyy-MM-dd
HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO" {code}
We discovered that when the value of log.layout.pattern contains spaces, the
command is incorrectly truncated, causing the content after the space to be
unrecognized, and therefore the Flink job cannot be launched in the Yarn
container.
{code:java}
{HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO"{code}
Therefore, we suggest using single quotes (') instead of double quotes (") to
wrap the values of parameters in the {{getLogBackCommand}} method and
{{getLog4jCommand}} method of the {{YarnLogConfigUtil}} class.
!image-2024-07-25-21-52-26-973.png|width=843,height=506!!image-2024-07-25-21-51-50-072.png|width=417,height=218!
> When running Flink on Yarn, if the log file path contains a space, the
> startup will fail.
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-35898
> URL: https://issues.apache.org/jira/browse/FLINK-35898
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.20.0
> Reporter: yiheng tang
> Priority: Minor
> Attachments: image-2024-07-25-21-51-50-072.png,
> image-2024-07-25-21-52-26-973.png
>
>
> We use Yarn to launch Flink jobs and try to add custom parameters to
> configure the log output format in the YarnLogConfigUtil.getLog4jCommand
> method. However, when the parameter value contains spaces, it causes the
> startup to fail.
> The start command in the Yarn container is as follows, omitting some
> irrelevant parameters.
> {code:java}
> /bin/bash -c "$JAVA_HOME/bin/java
> -Dlog.file="/data08/yarn/userlogs/application_1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log"
> -Dlog4j.configuration=file.properties
> -Dlog4j.configurationFile=file.properties
> -Dlog4j2.isThreadContextMapInheritable=true
> -Dlog.layout.pattern="%d{yyyy-MM-dd HH:mm} %-5p %-60c %-60t %x - %m%n %ex"
> -Dlog.level=INFO" {code}
> We discovered that when the value of log.layout.pattern contains spaces, the
> command is incorrectly truncated, causing the content after the space to be
> unrecognized, and therefore the Flink job cannot be launched in the Yarn
> container.
> {code:java}
> {HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO"{code}
> Therefore, we suggest using single quotes (') instead of double quotes (") to
> wrap the values of parameters in the {{getLogBackCommand}} method and
> {{getLog4jCommand}} method of the {{YarnLogConfigUtil}} class.
> {code:java}
> private static String getLogBackCommand(final String logConfigFilePath) {
> final boolean hasLogback =
> logConfigFilePath.endsWith(CONFIG_FILE_LOGBACK_NAME);
> if (!hasLogback) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlogback.configurationFile=file:" +
> CONFIG_FILE_LOGBACK_NAME)
> .toString();
> } private static String getLog4jCommand(final String
> logConfigFilePath) {
> final boolean hasLog4j =
> logConfigFilePath.endsWith(CONFIG_FILE_LOG4J_NAME);
> if (!hasLog4j) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlog4j.configuration=file:" +
> CONFIG_FILE_LOG4J_NAME)
> .append(" -Dlog4j.configurationFile=file:" +
> CONFIG_FILE_LOG4J_NAME)
> .toString();
> }
> } {code}
> ---
> !image-2024-07-25-21-52-26-973.png|width=843,height=506!!image-2024-07-25-21-51-50-072.png|width=417,height=218!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
