Hyukjin Kwon created SPARK-61968:
------------------------------------

             Summary: Make SparkSession.emptyDataFrame take PySpark's required 
schema
                 Key: SPARK-61968
                 URL: https://issues.apache.org/jira/browse/SPARK-61968
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0's SparkSession.emptyDataFrame(self, schema) is 
createDataFrame([], schema)
(python/pyspark/sql/connect/session.py:798-799), so the schema is required, and 
a schema of None
fails as createDataFrame([], None) does. The drop-in's emptyDataFrame is 
signature (schema=None),
"a permissive superset" in its code comment (crates/pyspark-rs/src/session.rs): 
without a schema,
or with None, it returns the core's DataFrame of no rows and no columns. The 
signature that
inspect shows differs too, and nothing documents the difference.

Repro (master 81751efd, against a Spark 4.3.0 server)
  >>> import inspect
  >>> inspect.signature(spark.emptyDataFrame)
  <Signature (schema=None)>
  >>> spark.emptyDataFrame()
  DataFrame[]
  >>> spark.emptyDataFrame(None)
  DataFrame[]

Expected (as with pyspark-client 4.3.0)
  >>> inspect.signature(spark.emptyDataFrame)
  <Signature (schema: Union[pyspark.sql.types.StructType, str]) -> 
'ParentDataFrame'>
  >>> spark.emptyDataFrame()
  TypeError: SparkSession.emptyDataFrame() missing 1 required positional 
argument: 'schema'
  >>> spark.emptyDataFrame(None)
  PySparkValueError: [CANNOT_INFER_EMPTY_SCHEMA] Can not infer schema from an 
empty dataset.
The drop-in's signature is then (schema), without PySpark's annotations, as its 
other native
methods' (docs/compatibility.md). A schema, a StructType or a DDL string, gives 
the DataFrame of
no rows of the schema, as before.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to