[
https://issues.apache.org/jira/browse/SPARK-61962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61962.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 837
[https://github.com/apache/spark-connect-rust/pull/837]
> Read the output type of a UDF and the schema of to() and of map DataFrames
> when the plan is built, bind the columns of map UDFs by name, and match
> PySpark's transformWithState, __getitem__, signature, randomSplit and Window
> details
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61962
> URL: https://issues.apache.org/jira/browse/SPARK-61962
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> A round-3 review of the Python DataFrame API and the map functions against
> pyspark-client 4.3.0:
> 1. A UDF's output type, and the cached schema of a map DataFrame, were fixed
> at the call, while the type pickled with the function was read when the plan
> was built. PySpark builds both from the same object at each build
> (connect/expressions.py:752-766), and caches that object as the map
> DataFrame's schema (connect/dataframe.py:2291-2292, group.py:330-331). So
> after `rt = StructType([...]); c = udf(f, rt)("id"); rt.add("b",
> LongType())`, PySpark's UDF returned the new field, where the drop-in raised
> ARROW_TYPE_MISMATCH or STRUCT_ARRAY_LENGTH_MISMATCH; and
> mapInPandas/applyInPandas silently dropped a field added after the call.
> df.to(schema) snapshotted its schema the same way
> (connect/dataframe.py:2079-2086).
> 2. Map and group-map UDFs bound an input column named "*" or "x.*" as a star
> (connect/udf.py:198-206 uses ColumnReference).
> 3. transformWithState and transformWithStateInPandas refused a
> non-GroupedData initialState with PyO3's TypeError, where PySpark raises the
> AttributeError of `_df` or SESSION_NOT_SAME (connect/group.py:406, 458). They
> also read the input's columns before checking the output type, where PySpark
> raises NOT_EXPECTED_TYPE first.
> 4. DataFrame.__getitem__ ignored PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1
> (connect/dataframe.py:1918-1930).
> 5. Signature defaults showed Ellipsis: unionByName, SparkSession.range,
> TableValuedFunction.range and StreamingQuery.explain.
> 6. randomSplit took only a Sequence of weights, where PySpark iterates any
> iterable, and Decimal weights raise its TypeError.
> 7. Window lacked _PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD, Window()
> could not be instantiated, and pyspark.sql.window.__all__ listed FrameBound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]