Hyukjin Kwon created SPARK-61968:
------------------------------------
Summary: Make SparkSession.emptyDataFrame take PySpark's required
schema
Key: SPARK-61968
URL: https://issues.apache.org/jira/browse/SPARK-61968
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0's SparkSession.emptyDataFrame(self, schema) is
createDataFrame([], schema)
(python/pyspark/sql/connect/session.py:798-799), so the schema is required, and
a schema of None
fails as createDataFrame([], None) does. The drop-in's emptyDataFrame is
signature (schema=None),
"a permissive superset" in its code comment (crates/pyspark-rs/src/session.rs):
without a schema,
or with None, it returns the core's DataFrame of no rows and no columns. The
signature that
inspect shows differs too, and nothing documents the difference.
Repro (master 81751efd, against a Spark 4.3.0 server)
>>> import inspect
>>> inspect.signature(spark.emptyDataFrame)
<Signature (schema=None)>
>>> spark.emptyDataFrame()
DataFrame[]
>>> spark.emptyDataFrame(None)
DataFrame[]
Expected (as with pyspark-client 4.3.0)
>>> inspect.signature(spark.emptyDataFrame)
<Signature (schema: Union[pyspark.sql.types.StructType, str]) ->
'ParentDataFrame'>
>>> spark.emptyDataFrame()
TypeError: SparkSession.emptyDataFrame() missing 1 required positional
argument: 'schema'
>>> spark.emptyDataFrame(None)
PySparkValueError: [CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an
empty dataset.
The drop-in's signature is then (schema), without PySpark's annotations, as its
other native
methods' (docs/compatibility.md). A schema, a StructType or a DDL string, gives
the DataFrame of
no rows of the schema, as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]