[
https://issues.apache.org/jira/browse/SPARK-61414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61414:
-----------------------------------
Labels: pull-request-available (was: )
> Send a limit of one row over an empty projection in DataFrame::is_empty of
> the Rust API, as PySpark does
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61414
> URL: https://issues.apache.org/jira/browse/SPARK-61414
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> DataFrame::is_empty of the Rust API, and so DataFrame.isEmpty of the Python
> client of spark-connect-rust, sends a global count aggregate over the limit:
> Aggregate(count(1)) over Limit(1). PySpark's isEmpty is
> {{len(self.select().take(1)) == 0}}: it sends Limit(1) over a Project with no
> expressions and checks whether a row comes back. The answers are the same;
> the plan, and so what the server runs and returns, is not.
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> print(spark.range(3).isEmpty())
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback, the plan of the
> request:
> * pyspark-client 4.3.0: Limit(limit=1, input=Project(expressions=[],
> input=Range(0, 3, 1))). The result is False, from the row that comes back.
> * spark-connect-rust master 125654b: Aggregate(group_type=Groupby,
> aggregate_expressions=[count(1)], input=Limit(limit=1, input=Range(0, 3,
> 1))). The result is False, from the count.
> From source, at 125654b: DataFrame::is_empty is
> {{self.limit(1).count().map(|c| c == 0)}}
> (crates/spark-connect/src/dataframe.rs:1315-1317), and DataFrame::count
> pushes a {{count(1)}} aggregate
> (crates/spark-connect/src/dataframe.rs:1052-1070). The Python client calls
> DataFrame::is_empty (crates/pyspark-rs/src/dataframe.rs:2631-2635). PySpark's
> isEmpty is in python/pyspark/sql/connect/dataframe.py.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]