srowen commented on a change in pull request #26027: [SPARK-24540][SQL] Support
for multiple character delimiter in Spark CSV read
URL: https://github.com/apache/spark/pull/26027#discussion_r334243092
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
##########
@@ -188,6 +190,30 @@ class CSVSuite extends QueryTest with SharedSparkSession
with TestCsvData {
verifyCars(cars, withHeader = true)
}
+ test("test with tab delimiter and double quote") {
+ val cars = spark.read
+ .options(Map("quote" -> "\"", "delimiter" -> """\t""", "header" ->
"true"))
+ .csv(testFile(carsTsvFile))
+
+ verifyCars(cars, numFields = 6, withHeader = true, checkHeader = false)
+ }
+
+ test("SPARK-24540: test with multiple character delimiter (comma space)") {
+ val cars = spark.read
+ .options(Map("quote" -> "\'", "delimiter" -> ", ", "header" -> "true"))
+ .csv(testFile(carsMultiCharDelimitedFile))
+
+ verifyCars(cars, withHeader = true)
+ }
+
+ test("SPARK-24540: test with multiple (crazy) character delimiter") {
+ val cars = spark.read
+ .options(Map("quote" -> "\'", "delimiter" -> """_/-\\_""", "header" ->
"true"))
Review comment:
Hm, wait, one more possible issue. This specifies the delimiter as string
`_/-\_`, which should then be unescaped to `_/-_`, which isn't what your test
case expects. The test passes because I don't think it checks anything but
year? Hm, I actually trying to figure out how it works at all in this case, as
the CSV file uses `_/-\_` as the delimiter. @jeff303 am I missing something or
is there a test issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
