Hyukjin Kwon created SPARK-61863:
------------------------------------

             Summary: Let other threads and Ctrl-C interrupt the native 
conversion of createDataFrame, and follow PySpark's pyarrow width errors and 
the cache threshold of the Rust API
                 Key: SPARK-61863
                 URL: https://issues.apache.org/jira/browse/SPARK-61863
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-61772 and SPARK-61837 for createDataFrame:
- The native conversion of local data (rows, tuples, dicts, the pandas row 
path, inference) runs with no Python checkpoint and no signal check, so with 
1.2M rows another Python thread pauses 2.3 to 3.9 s and Ctrl-C waits 3.3 to 5 
s; PySpark's conversion is Python, which other threads and signals interrupt 
(tens of ms). SPARK-61837 added such checkpoints to the result conversion only.
- A pyarrow table whose fields differ from a struct schema raises 
AXIS_LENGTH_MISMATCH of a pre-check, where PySpark raises PyArrow's ValueError 
of its cast; a pandas frame of no columns under StructType([]) gives [Row(), 
Row()] where PySpark raises AXIS_LENGTH_MISMATCH.
- The Rust API's create_dataframe ignores the server's 
localRelationCacheThreshold below 64 KiB, unlike create_dataframe_from_arrow, 
the Python binding and PySpark.
- Comments still say that the pandas helpers run before or avoid a Config 
request.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to