Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20004#discussion_r159010802
--- Diff: python/pyspark/sql/streaming.py ---
@@ -621,6 +621,10 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
``spark.sql.columnNameOfCorruptRecord``.
:param multiLine: parse one record, which may span multiple lines.
If None is
set, it uses the default value, ``false``.
+ :param charToEscapeQuoteEscaping: sets a single character used for
escaping the escape for
+ the quote character. If None is
set, the default value is
+ escape character when escape and
quote characters are
+ different, ``\0`` otherwise..
--- End diff --
Since we are relying on the external behavior to enforce this logics, we
have two options:
- enforce this in our logics and add unit test cases
- add a test case to ensure the future parser package upgrade will not
break this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
