Hyukjin Kwon created SPARK-61962:
------------------------------------
Summary: Read the output type of a UDF and the schema of to() and
of map DataFrames when the plan is built, bind the columns of map UDFs by name,
and match PySpark's transformWithState, __getitem__, signature, randomSplit and
Window details
Key: SPARK-61962
URL: https://issues.apache.org/jira/browse/SPARK-61962
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A round-3 review of the Python DataFrame API and the map functions against
pyspark-client 4.3.0:
1. A UDF's output type, and the cached schema of a map DataFrame, were fixed at
the call, while the type pickled with the function was read when the plan was
built. PySpark builds both from the same object at each build
(connect/expressions.py:752-766), and caches that object as the map DataFrame's
schema (connect/dataframe.py:2291-2292, group.py:330-331). So after `rt =
StructType([...]); c = udf(f, rt)("id"); rt.add("b", LongType())`, PySpark's
UDF returned the new field, where the drop-in raised ARROW_TYPE_MISMATCH or
STRUCT_ARRAY_LENGTH_MISMATCH; and mapInPandas/applyInPandas silently dropped a
field added after the call. df.to(schema) snapshotted its schema the same way
(connect/dataframe.py:2079-2086).
2. Map and group-map UDFs bound an input column named "*" or "x.*" as a star
(connect/udf.py:198-206 uses ColumnReference).
3. transformWithState and transformWithStateInPandas refused a non-GroupedData
initialState with PyO3's TypeError, where PySpark raises the AttributeError of
`_df` or SESSION_NOT_SAME (connect/group.py:406, 458). They also read the
input's columns before checking the output type, where PySpark raises
NOT_EXPECTED_TYPE first.
4. DataFrame.__getitem__ ignored PYSPARK_VALIDATE_COLUMN_NAME_LEGACY=1
(connect/dataframe.py:1918-1930).
5. Signature defaults showed Ellipsis: unionByName, SparkSession.range,
TableValuedFunction.range and StreamingQuery.explain.
6. randomSplit took only a Sequence of weights, where PySpark iterates any
iterable, and Decimal weights raise its TypeError.
7. Window lacked _PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD, Window() could
not be instantiated, and pyspark.sql.window.__all__ listed FrameBound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]