[ 
https://issues.apache.org/jira/browse/SPARK-61413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61413.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 519
[https://github.com/apache/spark-connect-rust/pull/519]

> Return a generator from DataFrame.toLocalIterator in the Python client, as 
> PySpark does
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-61413
>                 URL: https://issues.apache.org/jira/browse/SPARK-61413
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Assignee: Tim Meehan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> DataFrame.toLocalIterator of the Python client returns a LocalRowIterator, an 
> object of a Rust class, where PySpark's is a generator function that returns 
> a generator. The rows are the same, and since SPARK-61160 the iterator ends 
> its execution on close() as a generator does, but it is not a generator: 
> isinstance(rows, types.GeneratorType) and inspect.isgenerator(rows) are 
> False, and it has none of the generator attributes and methods.
> {code:python}
> import inspect, types
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> rows = spark.range(3).toLocalIterator()
> print(type(rows).__name__, isinstance(rows, types.GeneratorType), 
> inspect.isgenerator(rows))
> print(rows.__qualname__)
> rows.throw(ValueError("stop"))
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: {{generator True True}}, the qualified name 
> {{DataFrame.toLocalIterator}}, and {{throw}} raises the ValueError and leaves 
> the generator closed (inspect.getgeneratorstate is GEN_CLOSED).
> * the Python client at spark-connect-rust master 125654b: {{LocalRowIterator 
> False False}}; {{__qualname__}}, {{gi_frame}}, {{gi_running}}, {{send}} and 
> {{throw}} raise AttributeError, so inspect.getgeneratorstate fails too.
> What does not differ: nothing is sent before the first next() in either 
> client, an error of the execution is raised by the first next() and the 
> iteration ends after it, and close(), contextlib.closing and exhaustion end 
> the iterator the same way. throw() is a new case: it must end the execution 
> too, as PySpark's does.
> From source, at 125654b: DataFrame.toLocalIterator returns a 
> PyLocalRowIterator (crates/pyspark-rs/src/dataframe.rs:2303-2307), a pyclass 
> named LocalRowIterator (dataframe.rs:4535) that implements __iter__, __next__ 
> and close only.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to