Hyukjin Kwon created SPARK-61497:
------------------------------------
Summary: Raise for a str that UTF-8 cannot encode in the options
of the stream reader, and set the wholetext default of text(), in the Python
client as PySpark does
Key: SPARK-61497
URL: https://issues.apache.org/jira/browse/SPARK-61497
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-ups of a post-hoc review of SPARK-61475, checked against PySpark 4.3.0:
- The stream reader's option() and its format methods (for example
text(lineSep=...), csv(sep=...), and the path option of load()) sent a str that
UTF-8 cannot encode, such as a lone surrogate from a surrogateescape-decoded
path, as U+FFFD three times: another text, such as another path. SPARK-61475
made them convert through a lossy str(). The batch reader, readStream.options()
and load(**options) raise protobuf's ValueError, as PySpark does.
- text() of the batch and stream readers took wholetext=None, so an earlier
wholetext option was kept. PySpark's text(paths, wholetext=False, ...) sets its
default on every call through _set_opts. So spark.read.option("wholetext",
True).text(path) reads one row per line in PySpark, and one row per file in the
Python client. readStream.text(path) also leaves wholetext=False in the
reader's options in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]