Hyukjin Kwon created SPARK-61911:
------------------------------------
Summary: Raise the remaining argument errors where PySpark raises
them, and send MERGE assignments in their order
Key: SPARK-61911
URL: https://issues.apache.org/jira/browse/SPARK-61911
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61880 from a round-2 review against PySpark 4.3.0:
- zipWithIndex converts its column name at the call, where PySpark sets it at
plan build (bytes work there), and zipWithIndex and toJSON send no col and
alias origins.
- applyInPandasWithState, transformWithStateInPandas and transformWithState
convert their modes at the call; transformWithState sends its output type
parsed, and both TWS methods drop event_time_column_name "".
- The avro and protobuf functions take their schema, message name and
descriptor as Rust types, where PySpark wraps them in lit().
- mapInPandas and mapInArrow take only a bool barrier at the call, and send the
resource profile's create command at the call, where PySpark sets the flag and
reads the profile at plan build.
- nearestByJoin, readStream.name, Column and SparkSession __getattr__, the
connect Builder.master, registerJavaFunction, udtf.register, udf.register(None,
...), interruptTag and cogroup(5) raise other errors than PySpark's, or refuse
what PySpark takes.
- MergeIntoWriter checks its arguments at the call, and sends its assignments,
and raises their errors, in a random HashMap order, where PySpark keeps the
dict's order.
- An UNSUPPORTED_OPERATION message loses a lone surrogate; spark.client's
copies and pickles fail with other errors than PySpark's; StorageLevel does not
pickle at protocols 0 and 1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]