[ 
https://issues.apache.org/jira/browse/SPARK-61493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61493:
------------------------------------

    Assignee: Hyukjin Kwon

> Ignore the prefetchPartitions of toLocalIterator, give its generator the code 
> name toLocalIterator, and do not check the session IDs of artifact responses, 
> as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61493
>                 URL: https://issues.apache.org/jira/browse/SPARK-61493
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Follow-ups of a post-hoc review of SPARK-61413 and SPARK-61410, checked 
> against PySpark 4.3.0:
> - DataFrame.toLocalIterator of the Python client takes prefetchPartitions as 
> a bool, so a value such as 1, None or "x" raises a TypeError at the call. 
> PySpark Connect's toLocalIterator(self, prefetchPartitions: bool = False) 
> ignores the value and returns the rows.
> - The generator that toLocalIterator returns has the code name _local_rows, 
> which gi_code, tracebacks and inspect show. PySpark's generator has the code 
> name toLocalIterator (qualified name DataFrame.toLocalIterator).
> - The Rust client checks the server-side session ID of the ArtifactStatus and 
> AddArtifacts responses. So when spark.addArtifacts(...), or a cache of 
> artifacts, is the first call after a server restart, it raises an assertion 
> error and closes the client. PySpark's ArtifactManager does not check the IDs 
> of these responses (and its requests carry no observed server-side session 
> ID): the call returns, and the next call of the client raises.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to