[ 
https://issues.apache.org/jira/browse/SPARK-61414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61414:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send a limit of one row over an empty projection in DataFrame::is_empty of 
> the Rust API, as PySpark does
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61414
>                 URL: https://issues.apache.org/jira/browse/SPARK-61414
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> DataFrame::is_empty of the Rust API, and so DataFrame.isEmpty of the Python 
> client of spark-connect-rust, sends a global count aggregate over the limit: 
> Aggregate(count(1)) over Limit(1). PySpark's isEmpty is 
> {{len(self.select().take(1)) == 0}}: it sends Limit(1) over a Project with no 
> expressions and checks whether a row comes back. The answers are the same; 
> the plan, and so what the server runs and returns, is not.
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> print(spark.range(3).isEmpty())
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback, the plan of the 
> request:
> * pyspark-client 4.3.0: Limit(limit=1, input=Project(expressions=[], 
> input=Range(0, 3, 1))). The result is False, from the row that comes back.
> * spark-connect-rust master 125654b: Aggregate(group_type=Groupby, 
> aggregate_expressions=[count(1)], input=Limit(limit=1, input=Range(0, 3, 
> 1))). The result is False, from the count.
> From source, at 125654b: DataFrame::is_empty is 
> {{self.limit(1).count().map(|c| c == 0)}} 
> (crates/spark-connect/src/dataframe.rs:1315-1317), and DataFrame::count 
> pushes a {{count(1)}} aggregate 
> (crates/spark-connect/src/dataframe.rs:1052-1070). The Python client calls 
> DataFrame::is_empty (crates/pyspark-rs/src/dataframe.rs:2631-2635). PySpark's 
> isEmpty is in python/pyspark/sql/connect/dataframe.py.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to