Hyukjin Kwon created SPARK-61497:
------------------------------------

             Summary: Raise for a str that UTF-8 cannot encode in the options 
of the stream reader, and set the wholetext default of text(), in the Python 
client as PySpark does
                 Key: SPARK-61497
                 URL: https://issues.apache.org/jira/browse/SPARK-61497
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-ups of a post-hoc review of SPARK-61475, checked against PySpark 4.3.0:
- The stream reader's option() and its format methods (for example 
text(lineSep=...), csv(sep=...), and the path option of load()) sent a str that 
UTF-8 cannot encode, such as a lone surrogate from a surrogateescape-decoded 
path, as U+FFFD three times: another text, such as another path. SPARK-61475 
made them convert through a lossy str(). The batch reader, readStream.options() 
and load(**options) raise protobuf's ValueError, as PySpark does.
- text() of the batch and stream readers took wholetext=None, so an earlier 
wholetext option was kept. PySpark's text(paths, wholetext=False, ...) sets its 
default on every call through _set_opts. So spark.read.option("wholetext", 
True).text(path) reads one row per line in PySpark, and one row per file in the 
Python client. readStream.text(path) also leaves wholetext=False in the 
reader's options in PySpark.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to