Tim Meehan created SPARK-61413:
----------------------------------
Summary: Return a generator from DataFrame.toLocalIterator in the
Python client, as PySpark does
Key: SPARK-61413
URL: https://issues.apache.org/jira/browse/SPARK-61413
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
DataFrame.toLocalIterator of the Python client returns a LocalRowIterator, an
object of a Rust class, where PySpark's is a generator function that returns a
generator. The rows are the same, and since SPARK-61160 the iterator ends its
execution on close() as a generator does, but it is not a generator:
isinstance(rows, types.GeneratorType) and inspect.isgenerator(rows) are False,
and it has none of the generator attributes and methods.
{code:python}
import inspect, types
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
rows = spark.range(3).toLocalIterator()
print(type(rows).__name__, isinstance(rows, types.GeneratorType),
inspect.isgenerator(rows))
print(rows.__qualname__)
rows.throw(ValueError("stop"))
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: {{generator True True}}, the qualified name
{{DataFrame.toLocalIterator}}, and {{throw}} raises the ValueError and leaves
the generator closed (inspect.getgeneratorstate is GEN_CLOSED).
* the Python client at spark-connect-rust master 125654b: {{LocalRowIterator
False False}}; {{__qualname__}}, {{gi_frame}}, {{gi_running}}, {{send}} and
{{throw}} raise AttributeError, so inspect.getgeneratorstate fails too.
What does not differ: nothing is sent before the first next() in either client,
an error of the execution is raised by the first next() and the iteration ends
after it, and close(), contextlib.closing and exhaustion end the iterator the
same way. throw() is a new case: it must end the execution too, as PySpark's
does.
>From source, at 125654b: DataFrame.toLocalIterator returns a
>PyLocalRowIterator (crates/pyspark-rs/src/dataframe.rs:2303-2307), a pyclass
>named LocalRowIterator (dataframe.rs:4535) that implements __iter__, __next__
>and close only.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]