Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1724#discussion_r15889526
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
---
@@ -148,4 +148,29 @@ object YarnSparkHadoopUtil {
}
}
+ /**
+ * Escapes a string for inclusion in a command line executed by Yarn.
Yarn executes commands
+ * using `bash -c "command arg1 arg2"` and that means plain quoting
doesn't really work. The
+ * argument is enclosed in single quotes and some key characters are
escaped.
+ *
+ * @param arg A single argument.
+ * @return Argument quoted for execution via Yarn's generated shell
script.
+ */
+ def escapeForShell(arg: String): String = {
+ if (arg != null) {
+ val escaped = new StringBuilder("'")
+ for (i <- 0 to arg.length() - 1) {
+ arg.charAt(i) match {
+ case '$' => escaped.append("\\$")
+ case '"' => escaped.append("\\\"")
+ case '\'' => escaped.append("'\\\"'\\\"'")
--- End diff --
Yeah, this is the tricky one. Escaping single quotes inside a single-quoted
string does not work.
So what you have to do is close the previous string (remember the whole
thing is wrapped in single quotes), and start a new string, delimited by double
quotes, with a single single quote in it. So basically, for a string containing
a single single quote, you're concatenating three different strings:
- An empty string at the start: ''
- The single quote, wrapped in double quotes: "'"
- An empty string at the end: ''
And since all this is already inside double quotes in the bash script
itself, you need to also escape the double quotes. Fun.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
