Hyukjin Kwon created SPARK-61483:
------------------------------------

             Summary: Send the bytes of binary literal subclasses, corr and cov 
without a limit, and check the Arrow schema of createDataFrame depth first, in 
the Python client as PySpark does
                 Key: SPARK-61483
                 URL: https://issues.apache.org/jira/browse/SPARK-61483
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-ups of the review of SPARK-61410, SPARK-61414, SPARK-61434 and 
SPARK-61435, each verified against PySpark 4.3.0:

1. lit() and the typed literals of the Python client (lit, isin, when, 
otherwise, getItem, create_map, typed arrays, sql args) read the raw buffer of 
a bytes or bytearray subclass, where PySpark sends bytes(value).
2. While raising an error of a client that the server-side session change 
closed, the client still sent a Config to read the stack-trace setting; PySpark 
sends nothing there.
3. corr and cov sent Limit(1) over their StatCorr/StatCov relation; PySpark 
sends the relation as it is. The Python client's isEmpty skipped the 
binaryAsBytes Config that PySpark's take -> collect reads after the execution.
4. createDataFrame of local rows checked the Arrow schema in two passes instead 
of depth first, as PySpark's to_arrow_type does, so a schema with an 
unsupported type before a non-nullable null type raised ValueError where 
PySpark raises PySparkTypeError; and its "not a sequence" check accepted 
generators, iterators, dict_keys, dict_items, frozensets and mappingproxies, 
which PyArrow rejects with ArrowTypeError.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to