[
https://issues.apache.org/jira/browse/SPARK-61968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61968:
-----------------------------------
Labels: pull-request-available (was: )
> Make SparkSession.emptyDataFrame take PySpark's required schema
> ---------------------------------------------------------------
>
> Key: SPARK-61968
> URL: https://issues.apache.org/jira/browse/SPARK-61968
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's SparkSession.emptyDataFrame(self, schema) is
> createDataFrame([], schema)
> (python/pyspark/sql/connect/session.py:798-799), so the schema is required,
> and a schema of None
> fails as createDataFrame([], None) does. The drop-in's emptyDataFrame is
> signature (schema=None),
> "a permissive superset" in its code comment
> (crates/pyspark-rs/src/session.rs): without a schema,
> or with None, it returns the core's DataFrame of no rows and no columns. The
> signature that
> inspect shows differs too, and nothing documents the difference.
> Repro (master 81751efd, against a Spark 4.3.0 server)
> >>> import inspect
> >>> inspect.signature(spark.emptyDataFrame)
> <Signature (schema=None)>
> >>> spark.emptyDataFrame()
> DataFrame[]
> >>> spark.emptyDataFrame(None)
> DataFrame[]
> Expected (as with pyspark-client 4.3.0)
> >>> inspect.signature(spark.emptyDataFrame)
> <Signature (schema: Union[pyspark.sql.types.StructType, str]) ->
> 'ParentDataFrame'>
> >>> spark.emptyDataFrame()
> TypeError: SparkSession.emptyDataFrame() missing 1 required positional
> argument: 'schema'
> >>> spark.emptyDataFrame(None)
> PySparkValueError: [CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an
> empty dataset.
> The drop-in's signature is then (schema), without PySpark's annotations, as
> its other native
> methods' (docs/compatibility.md). A schema, a StructType or a DDL string,
> gives the DataFrame of
> no rows of the schema, as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]