[
https://issues.apache.org/jira/browse/YARN-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anne Isabelle Macedo updated YARN-11811:
----------------------------------------
Description:
TL;DR the regex for USER_MOUNT_PATTERN [1] doesn't trim leading quotes if they
don't get sanitized somehow from a spark-submit command.
Context: we're trying to use the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS conf on a
Hadoop cluster provisioned by AWS EMR and, when the value for this variable is
provided with quotes (such as '/tmp:/src/container'), we get a cryptic error
message such as this one on container-launch:
{{Exception message: Mount must be a localized resource: '/tmp/}}
(notice the leading quote on the path)
I don't really know if this is a failure on AWS api for sanitizing these
quotes, but it seems that, if they make their way to container launch, the
variable's content will be used with this leading quote - thus provoking some
unexpected behavior.
However, without these quotes, I see errors such as:
{{{}Shell error output: Configuration does not allow mount{}}}{{{{}}{}}}
with a path that wasn't the one I provided in the variable.
I checked on regex101.com and it seems that the USER_MOUNT_PATTERN won't
actually ignore quotes, meaning that if they make it to hadoop they will be
part of the regex match groups and variable's content.
[1]
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java#L74]
was:
TL;DR the regex for USER_MOUNT_PATTERN [1] doesn't trim leading quotes if they
don't get sanitized somehow from a spark-submit command.
Context: we're trying to use the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS conf on a
Hadoop cluster provisioned by AWS EMR and, when the value for this variable is
provided with quotes (such as '/tmp:/src/container'), we get a cryptic error
message such as this one on container-launch:
{{Exception message: Mount must be a localized resource: '/tmp/}}
I don't really know if this is a failure on AWS api for sanitizing these
quotes, but it seems that, if they make their way to container launch, the
variable's content will be used with this leading quote - thus provoking some
unexpected behavior.
However, without these quotes, I see errors such as:
{{{}Shell error output: Configuration does not allow mount{}}}{{{}{}}}
with a path that wasn't the one I provided in the variable.
I checked on regex101.com and it seems that the USER_MOUNT_PATTERN won't
actually ignore quotes, meaning that if they make it to hadoop they will be
part of the regex match groups and variable's content.
[1]
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java#L74
> YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS doesn't know how to deal with leading
> quotes
> ---------------------------------------------------------------------------------
>
> Key: YARN-11811
> URL: https://issues.apache.org/jira/browse/YARN-11811
> Project: Hadoop YARN
> Issue Type: Bug
> Components: container-executor
> Affects Versions: 3.4.0
> Environment: We're using EMR emr-7.6.0. I know that there might be
> differences between EMR and FOSS Apache Hadoop, but I believe the bug also
> exists on the FOSS version.
> Reporter: Anne Isabelle Macedo
> Priority: Minor
>
> TL;DR the regex for USER_MOUNT_PATTERN [1] doesn't trim leading quotes if
> they don't get sanitized somehow from a spark-submit command.
>
> Context: we're trying to use the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS conf on
> a Hadoop cluster provisioned by AWS EMR and, when the value for this variable
> is provided with quotes (such as '/tmp:/src/container'), we get a cryptic
> error message such as this one on container-launch:
> {{Exception message: Mount must be a localized resource: '/tmp/}}
> (notice the leading quote on the path)
>
> I don't really know if this is a failure on AWS api for sanitizing these
> quotes, but it seems that, if they make their way to container launch, the
> variable's content will be used with this leading quote - thus provoking some
> unexpected behavior.
> However, without these quotes, I see errors such as:
> {{{}Shell error output: Configuration does not allow mount{}}}{{{{}}{}}}
> with a path that wasn't the one I provided in the variable.
>
> I checked on regex101.com and it seems that the USER_MOUNT_PATTERN won't
> actually ignore quotes, meaning that if they make it to hadoop they will be
> part of the regex match groups and variable's content.
>
> [1]
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java#L74]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
