Hyukjin Kwon created SPARK-61911:
------------------------------------

             Summary: Raise the remaining argument errors where PySpark raises 
them, and send MERGE assignments in their order
                 Key: SPARK-61911
                 URL: https://issues.apache.org/jira/browse/SPARK-61911
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-61880 from a round-2 review against PySpark 4.3.0:
- zipWithIndex converts its column name at the call, where PySpark sets it at 
plan build (bytes work there), and zipWithIndex and toJSON send no col and 
alias origins.
- applyInPandasWithState, transformWithStateInPandas and transformWithState 
convert their modes at the call; transformWithState sends its output type 
parsed, and both TWS methods drop event_time_column_name "".
- The avro and protobuf functions take their schema, message name and 
descriptor as Rust types, where PySpark wraps them in lit().
- mapInPandas and mapInArrow take only a bool barrier at the call, and send the 
resource profile's create command at the call, where PySpark sets the flag and 
reads the profile at plan build.
- nearestByJoin, readStream.name, Column and SparkSession __getattr__, the 
connect Builder.master, registerJavaFunction, udtf.register, udf.register(None, 
...), interruptTag and cogroup(5) raise other errors than PySpark's, or refuse 
what PySpark takes.
- MergeIntoWriter checks its arguments at the call, and sends its assignments, 
and raises their errors, in a random HashMap order, where PySpark keeps the 
dict's order.
- An UNSUPPORTED_OPERATION message loses a lone surrogate; spark.client's 
copies and pickles fail with other errors than PySpark's; StorageLevel does not 
pickle at protocols 0 and 1.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to