[
https://issues.apache.org/jira/browse/FLINK-35898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869333#comment-17869333
]
Ferenc Csaky commented on FLINK-35898:
--------------------------------------
AFAIK what you describe cannot happen. The Yarn {{applicationId}} and
{{containerId}} are generated by Yarn and cannot contain whitespaces. I think
technically the directories that precede the {{applicationId}} dir and
configured by {{yarn.nodemanager.log-dirs}} may contain whitespaces, but the
Cloudera platform (which I have experience with) for example does not allow to
use such directory and that is in general an anti-pattern I think.
Can you can share what kind of configs are in use with the faulty deployment?
The original description does not contain a whitespace in the {{log.file}}
path, and neither does right now after the edits, so I am still not convinced
that the mentioned property causes the failure.
> When running Flink on Yarn, if the log file path contains a space, the
> startup will fail.
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-35898
> URL: https://issues.apache.org/jira/browse/FLINK-35898
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.20.0
> Reporter: yiheng tang
> Priority: Minor
> Attachments: image-2024-07-25-21-51-50-072.png,
> image-2024-07-25-21-52-26-973.png
>
>
> We use Yarn to launch Flink jobs and try to add custom parameters to
> configure the log output format in the YarnLogConfigUtil.getLog4jCommand
> method. However, when the parameter value contains spaces, it causes the
> startup to fail.
> The start command in the Yarn container is as follows, omitting some
> irrelevant parameters.
> {code:java}
> /bin/bash -c "$JAVA_HOME/bin/java
> -Dlog.file="/data08/yarn/userlogs/application_1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log"
> -Dlog4j.configuration=file.properties
> -Dlog4j.configurationFile=file.properties
> -Dlog4j2.isThreadContextMapInheritable=true
> -Dlog.layout.pattern="%d{yyyy-MM-dd HH:mm} %-5p %-60c %-60t %x - %m%n %ex"
> -Dlog.level=INFO" {code}
> We discovered that when the value of log.layout.pattern contains spaces, the
> command is incorrectly truncated, causing the content after the space to be
> unrecognized, and therefore the Flink job cannot be launched in the Yarn
> container.
> {code:java}
> {HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO"{code}
> Therefore, we suggest using single quotes (') instead of double quotes (") to
> wrap the values of parameters in the {{getLogBackCommand}} method and
> {{getLog4jCommand}} method of the {{YarnLogConfigUtil}} class.
> {code:java}
> private static String getLogBackCommand(final String logConfigFilePath) {
> final boolean hasLogback =
> logConfigFilePath.endsWith(CONFIG_FILE_LOGBACK_NAME);
> if (!hasLogback) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlogback.configurationFile=file:" +
> CONFIG_FILE_LOGBACK_NAME)
> .toString();
> } private static String getLog4jCommand(final String
> logConfigFilePath) {
> final boolean hasLog4j =
> logConfigFilePath.endsWith(CONFIG_FILE_LOG4J_NAME);
> if (!hasLog4j) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlog4j.configuration=file:" +
> CONFIG_FILE_LOG4J_NAME)
> .append(" -Dlog4j.configurationFile=file:" +
> CONFIG_FILE_LOG4J_NAME)
> .toString();
> }
> } {code}
> ---
> !image-2024-07-25-21-52-26-973.png|width=843,height=506!!image-2024-07-25-21-51-50-072.png|width=417,height=218!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
