[ 
https://issues.apache.org/jira/browse/SPARK-61968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61968.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 828
[https://github.com/apache/spark-connect-rust/pull/828]

> Make SparkSession.emptyDataFrame take PySpark's required schema
> ---------------------------------------------------------------
>
>                 Key: SPARK-61968
>                 URL: https://issues.apache.org/jira/browse/SPARK-61968
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's SparkSession.emptyDataFrame(self, schema) is 
> createDataFrame([], schema)
> (python/pyspark/sql/connect/session.py:798-799), so the schema is required, 
> and a schema of None
> fails as createDataFrame([], None) does. The drop-in's emptyDataFrame is 
> signature (schema=None),
> "a permissive superset" in its code comment 
> (crates/pyspark-rs/src/session.rs): without a schema,
> or with None, it returns the core's DataFrame of no rows and no columns. The 
> signature that
> inspect shows differs too, and nothing documents the difference.
> Repro (master 81751efd, against a Spark 4.3.0 server)
>   >>> import inspect
>   >>> inspect.signature(spark.emptyDataFrame)
>   <Signature (schema=None)>
>   >>> spark.emptyDataFrame()
>   DataFrame[]
>   >>> spark.emptyDataFrame(None)
>   DataFrame[]
> Expected (as with pyspark-client 4.3.0)
>   >>> inspect.signature(spark.emptyDataFrame)
>   <Signature (schema: Union[pyspark.sql.types.StructType, str]) -> 
> 'ParentDataFrame'>
>   >>> spark.emptyDataFrame()
>   TypeError: SparkSession.emptyDataFrame() missing 1 required positional 
> argument: 'schema'
>   >>> spark.emptyDataFrame(None)
>   PySparkValueError: [CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an 
> empty dataset.
> The drop-in's signature is then (schema), without PySpark's annotations, as 
> its other native
> methods' (docs/compatibility.md). A schema, a StructType or a DDL string, 
> gives the DataFrame of
> no rows of the schema, as before.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to