[
https://issues.apache.org/jira/browse/SPARK-61911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61911:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Raise the remaining argument errors where PySpark raises them, and send MERGE
> assignments in their order
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61911
> URL: https://issues.apache.org/jira/browse/SPARK-61911
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61880 from a round-2 review against PySpark 4.3.0:
> - zipWithIndex converts its column name at the call, where PySpark sets it at
> plan build (bytes work there), and zipWithIndex and toJSON send no col and
> alias origins.
> - applyInPandasWithState, transformWithStateInPandas and transformWithState
> convert their modes at the call; transformWithState sends its output type
> parsed, and both TWS methods drop event_time_column_name "".
> - The avro and protobuf functions take their schema, message name and
> descriptor as Rust types, where PySpark wraps them in lit().
> - mapInPandas and mapInArrow take only a bool barrier at the call, and send
> the resource profile's create command at the call, where PySpark sets the
> flag and reads the profile at plan build.
> - nearestByJoin, readStream.name, Column and SparkSession __getattr__, the
> connect Builder.master, registerJavaFunction, udtf.register,
> udf.register(None, ...), interruptTag and cogroup(5) raise other errors than
> PySpark's, or refuse what PySpark takes.
> - MergeIntoWriter checks its arguments at the call, and sends its
> assignments, and raises their errors, in a random HashMap order, where
> PySpark keeps the dict's order.
> - An UNSUPPORTED_OPERATION message loses a lone surrogate; spark.client's
> copies and pickles fail with other errors than PySpark's; StorageLevel does
> not pickle at protocols 0 and 1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]