Tim Meehan created SPARK-61292:
----------------------------------
Summary: Ask the server in DataFrame.isLocal of the Python client,
as PySpark does
Key: SPARK-61292
URL: https://issues.apache.org/jira/browse/SPARK-61292
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
PySpark's DataFrame.isLocal asks the server, with an AnalyzePlan request
(pyspark/sql/connect/dataframe.py:2053), and the server answers from the plan
it analyzes. The Python client of spark-connect-rust answers from its own plan
instead (DataFrame::is_local, crates/spark-connect/src/dataframe.rs:2244): only
a local relation that it built, or the result of a command, is local. So the
answers differ where the server's analysis makes a local relation of another
plan.
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
print(spark.sql("VALUES (1)").isLocal())
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: True
* spark-connect-rust master 8768cc4: False
Both give the same answer for spark.sql("SHOW DATABASES"), spark.sql("SELECT
1"), createDataFrame and range. PySpark caches the answer per DataFrame
(functools.cache), so a second call sends no request.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
