gdhuper commented on code in PR #41904:
URL: https://github.com/apache/spark/pull/41904#discussion_r1261777186
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala:
##########
@@ -253,16 +253,18 @@ class CSVOptions(
/**
* A string between two consecutive JSON records.
*/
- val lineSeparator: Option[String] = parameters.get(LINE_SEP).map { sep =>
- require(sep.nonEmpty, "'lineSep' cannot be an empty string.")
- // Intentionally allow it up to 2 for Window's CRLF although multiple
- // characters have an issue with quotes. This is intentionally
undocumented.
- require(sep.length <= 2, "'lineSep' can contain only 1 character.")
- if (sep.length == 2) logWarning("It is not recommended to set 'lineSep' " +
- "with 2 characters due to the limitation of supporting multi-char
'lineSep' within quotes.")
- sep
+ val lineSeparator: Option[String] = parameters.get(LINE_SEP) match {
+ case Some(sep) if sep != null =>
Review Comment:
@srowen I can see that it should be an error just like an empty value `''`.
However, one of the use cases where this might be helpful is where someone
might need to dynamically switch the value for `lineSep` from `None` to
something else. For example, code generation is one example that comes to mind,
where a base template with `lineSep: None` is used that can be replaced with
appropriate values as desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
