chanduhawk commented on a change in pull request #29516:
URL: https://github.com/apache/spark/pull/29516#discussion_r475231992
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
##########
@@ -220,7 +220,9 @@ class CSVOptions(
format.setQuote(quote)
format.setQuoteEscape(escape)
charToEscapeQuoteEscaping.foreach(format.setCharToEscapeQuoteEscaping)
- format.setComment(comment)
+ if (isCommentSet) {
Review comment:
I agree, but once the changes will be done then \u0000 won't be treated
as comment character. It will resolve this bug. But then default comment
character will be # as in univocity this is the default comment character. So
if my data row starts with # then will the row be processed now. If not then it
will break most of the existing jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
