Hyukjin Kwon created SPARK-61962:
------------------------------------

             Summary: Read the output type of a UDF and the schema of to() and 
of map DataFrames when the plan is built, bind the columns of map UDFs by name, 
and match PySpark's transformWithState, __getitem__, signature, randomSplit and 
Window details
                 Key: SPARK-61962
                 URL: https://issues.apache.org/jira/browse/SPARK-61962
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A round-3 review of the Python DataFrame API and the map functions against 
pyspark-client 4.3.0:

1. A UDF's output type, and the cached schema of a map DataFrame, were fixed at 
the call, while the type pickled with the function was read when the plan was 
built. PySpark builds both from the same object at each build 
(connect/expressions.py:752-766), and caches that object as the map DataFrame's 
schema (connect/dataframe.py:2291-2292, group.py:330-331). So after `rt = 
StructType([...]); c = udf(f, rt)("id"); rt.add("b", LongType())`, PySpark's 
UDF returned the new field, where the drop-in raised ARROW_TYPE_MISMATCH or 
STRUCT_ARRAY_LENGTH_MISMATCH; and mapInPandas/applyInPandas silently dropped a 
field added after the call. df.to(schema) snapshotted its schema the same way 
(connect/dataframe.py:2079-2086).

2. Map and group-map UDFs bound an input column named "*" or "x.*" as a star 
(connect/udf.py:198-206 uses ColumnReference).

3. transformWithState and transformWithStateInPandas refused a non-GroupedData 
initialState with PyO3's TypeError, where PySpark raises the AttributeError of 
`_df` or SESSION_NOT_SAME (connect/group.py:406, 458). They also read the 
input's columns before checking the output type, where PySpark raises 
NOT_EXPECTED_TYPE first.

4. DataFrame.__getitem__ ignored PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1 
(connect/dataframe.py:1918-1930).

5. Signature defaults showed Ellipsis: unionByName, SparkSession.range, 
TableValuedFunction.range and StreamingQuery.explain.

6. randomSplit took only a Sequence of weights, where PySpark iterates any 
iterable, and Decimal weights raise its TypeError.

7. Window lacked _PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD, Window() could 
not be instantiated, and pyspark.sql.window.__all__ listed FrameBound.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to