[
https://issues.apache.org/jira/browse/SPARK-61413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61413:
-----------------------------------
Labels: pull-request-available (was: )
> Return a generator from DataFrame.toLocalIterator in the Python client, as
> PySpark does
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61413
> URL: https://issues.apache.org/jira/browse/SPARK-61413
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> DataFrame.toLocalIterator of the Python client returns a LocalRowIterator, an
> object of a Rust class, where PySpark's is a generator function that returns
> a generator. The rows are the same, and since SPARK-61160 the iterator ends
> its execution on close() as a generator does, but it is not a generator:
> isinstance(rows, types.GeneratorType) and inspect.isgenerator(rows) are
> False, and it has none of the generator attributes and methods.
> {code:python}
> import inspect, types
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> rows = spark.range(3).toLocalIterator()
> print(type(rows).__name__, isinstance(rows, types.GeneratorType),
> inspect.isgenerator(rows))
> print(rows.__qualname__)
> rows.throw(ValueError("stop"))
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: {{generator True True}}, the qualified name
> {{DataFrame.toLocalIterator}}, and {{throw}} raises the ValueError and leaves
> the generator closed (inspect.getgeneratorstate is GEN_CLOSED).
> * the Python client at spark-connect-rust master 125654b: {{LocalRowIterator
> False False}}; {{__qualname__}}, {{gi_frame}}, {{gi_running}}, {{send}} and
> {{throw}} raise AttributeError, so inspect.getgeneratorstate fails too.
> What does not differ: nothing is sent before the first next() in either
> client, an error of the execution is raised by the first next() and the
> iteration ends after it, and close(), contextlib.closing and exhaustion end
> the iterator the same way. throw() is a new case: it must end the execution
> too, as PySpark's does.
> From source, at 125654b: DataFrame.toLocalIterator returns a
> PyLocalRowIterator (crates/pyspark-rs/src/dataframe.rs:2303-2307), a pyclass
> named LocalRowIterator (dataframe.rs:4535) that implements __iter__, __next__
> and close only.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]