Github user gerashegalov commented on a diff in the pull request:
https://github.com/apache/spark/pull/22213#discussion_r214237801
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -1144,6 +1144,46 @@ class SparkSubmitSuite
conf1.get(PY_FILES.key) should be (s"s3a://${pyFile.getAbsolutePath}")
conf1.get("spark.submit.pyFiles") should (startWith("/"))
}
+
+ test("handles natural line delimiters in --properties-file and --conf
uniformly") {
+ val delimKey = "spark.my.delimiter."
+ val LF = "\n"
+ val CR = "\r"
+
+ val leadingDelimKeyFromFile = s"${delimKey}leadingDelimKeyFromFile" ->
s"${LF}blah"
+ val trailingDelimKeyFromFile = s"${delimKey}trailingDelimKeyFromFile"
-> s"blah${CR}"
+ val infixDelimFromFile = s"${delimKey}infixDelimFromFile" ->
s"${CR}blah${LF}"
+ val nonDelimSpaceFromFile = s"${delimKey}nonDelimSpaceFromFile" -> "
blah\f"
--- End diff --
@jerryshao I try not to spend time on issues unrelated to our production
deployments. @steveloughran and this PR already pointed at the
`Properties#load` method documenting the format.
Line terminator characters can be included using `\r` and `\n` escape
sequences. Or you can encode any character using `\uxxxx`
In addition you can take a look at the file generated by this code:
```
#test whitespace
#Thu Aug 30 20:20:33 PDT 2018
spark.my.delimiter.nonDelimSpaceFromFile=\ blah\f
spark.my.delimiter.infixDelimFromFile=\rblah\n
spark.my.delimiter.trailingDelimKeyFromFile=blah\r
spark.my.delimiter.leadingDelimKeyFromFile=\nblah
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
