srowen commented on a change in pull request #29516:
URL: https://github.com/apache/spark/pull/29516#discussion_r475221331
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
##########
@@ -220,7 +220,9 @@ class CSVOptions(
format.setQuote(quote)
format.setQuoteEscape(escape)
charToEscapeQuoteEscaping.foreach(format.setCharToEscapeQuoteEscaping)
- format.setComment(comment)
+ if (isCommentSet) {
Review comment:
You are correct, but, this has never been a valid comment character, and
the flip side is the bug you describe: it's _always_ a comment character. I
think it's reasonable to fix as a bug. I don't think we need yet another
config, as I think it would be quite obscure to use this non-printing control
code for comments in a CSV file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
