Hyukjin Kwon created SPARK-61493:
------------------------------------

             Summary: Ignore the prefetchPartitions of toLocalIterator, give 
its generator the code name toLocalIterator, and do not check the session IDs 
of artifact responses, as PySpark does
                 Key: SPARK-61493
                 URL: https://issues.apache.org/jira/browse/SPARK-61493
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-ups of a post-hoc review of SPARK-61413 and SPARK-61410, checked against 
PySpark 4.3.0:
- DataFrame.toLocalIterator of the Python client takes prefetchPartitions as a 
bool, so a value such as 1, None or "x" raises a TypeError at the call. PySpark 
Connect's toLocalIterator(self, prefetchPartitions: bool = False) ignores the 
value and returns the rows.
- The generator that toLocalIterator returns has the code name _local_rows, 
which gi_code, tracebacks and inspect show. PySpark's generator has the code 
name toLocalIterator (qualified name DataFrame.toLocalIterator).
- The Rust client checks the server-side session ID of the ArtifactStatus and 
AddArtifacts responses. So when spark.addArtifacts(...), or a cache of 
artifacts, is the first call after a server restart, it raises an assertion 
error and closes the client. PySpark's ArtifactManager does not check the IDs 
of these responses (and its requests carry no observed server-side session ID): 
the call returns, and the next call of the client raises.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to