Github user ep1804 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20004#discussion_r157391233
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -482,6 +482,36 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils {
}
}
+ test("save csv with quote escaping, using escapeQuoteEscaping option") {
+ withTempPath { path =>
+ // when a string to be quoted ends with the escape character,
escapeQuoteEscaping is required
+ val df1 = Seq(
+ """You are "beautiful"""",
+ """Yes, \in the inside\""", // ends with the escape character '\\'
+ """AA\\BB""", // 2 escape char
+ """AA""BB""", // 2 quote char
+ """AA\"BB\""", // 1 escape char and 1 quote char
+ """AA\""BB\""", // 1 escape char and 2 quote char
+ """AA\\"BB\""" // 2 escape char and 1 quote char
+ ).toDF()
+
+ // escapeQuotes is true by default
+ // quote character is '\"' by default
+ // escape character is '\\' by default
+ df1.coalesce(1).write
+ .format("csv")
+ .option("escapeQuoteEscaping", "\\")
--- End diff --
It fails. As follows:
```
[info] == Results ==
[info] !== Correct Answer - 7 == == Spark Answer - 7 ==
[info] !struct<_c0:string> struct<value:string>
[info] [AA""BB] [AA""BB]
[info] ![AA\""BB"] [AA\""BB\]
[info] ![AA\"BB"] [AA\"BB\]
[info] ![AA\\"BB"] [AA\\"BB\]
[info] [AA\\BB] [AA\\BB]
[info] ![Yes, \in the inside"] [Yes, \in the inside\]
[info] [You are "beautiful"] [You are "beautiful"]
(QueryTest.scala:163)
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
