[ 
https://issues.apache.org/jira/browse/SPARK-61962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61962.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 837
[https://github.com/apache/spark-connect-rust/pull/837]

> Read the output type of a UDF and the schema of to() and of map DataFrames 
> when the plan is built, bind the columns of map UDFs by name, and match 
> PySpark's transformWithState, __getitem__, signature, randomSplit and Window 
> details
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61962
>                 URL: https://issues.apache.org/jira/browse/SPARK-61962
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> A round-3 review of the Python DataFrame API and the map functions against 
> pyspark-client 4.3.0:
> 1. A UDF's output type, and the cached schema of a map DataFrame, were fixed 
> at the call, while the type pickled with the function was read when the plan 
> was built. PySpark builds both from the same object at each build 
> (connect/expressions.py:752-766), and caches that object as the map 
> DataFrame's schema (connect/dataframe.py:2291-2292, group.py:330-331). So 
> after `rt = StructType([...]); c = udf(f, rt)("id"); rt.add("b", 
> LongType())`, PySpark's UDF returned the new field, where the drop-in raised 
> ARROW_TYPE_MISMATCH or STRUCT_ARRAY_LENGTH_MISMATCH; and 
> mapInPandas/applyInPandas silently dropped a field added after the call. 
> df.to(schema) snapshotted its schema the same way 
> (connect/dataframe.py:2079-2086).
> 2. Map and group-map UDFs bound an input column named "*" or "x.*" as a star 
> (connect/udf.py:198-206 uses ColumnReference).
> 3. transformWithState and transformWithStateInPandas refused a 
> non-GroupedData initialState with PyO3's TypeError, where PySpark raises the 
> AttributeError of `_df` or SESSION_NOT_SAME (connect/group.py:406, 458). They 
> also read the input's columns before checking the output type, where PySpark 
> raises NOT_EXPECTED_TYPE first.
> 4. DataFrame.__getitem__ ignored PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1 
> (connect/dataframe.py:1918-1930).
> 5. Signature defaults showed Ellipsis: unionByName, SparkSession.range, 
> TableValuedFunction.range and StreamingQuery.explain.
> 6. randomSplit took only a Sequence of weights, where PySpark iterates any 
> iterable, and Decimal weights raise its TypeError.
> 7. Window lacked _PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD, Window() 
> could not be instantiated, and pyspark.sql.window.__all__ listed FrameBound.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to