Hyukjin Kwon created SPARK-61190:
------------------------------------
Summary: Keep the CASE WHEN of a pickled when() Column, so when
and otherwise chain on it, in the Python client as PySpark does
Key: SPARK-61190
URL: https://issues.apache.org/jira/browse/SPARK-61190
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A when() Column that is pickled, cloudpickled or deep-copied came back as an
opaque expression (SPARK-61167), so .when() on it raised INVALID_WHEN_USAGE and
.otherwise() raised "otherwise() can only be applied on a Column previously
generated by when()". PySpark unpickles a CaseWhen, so both chain. The same
happened for a when() Column that a foreachBatch function captured, on the
Python worker of the server.
Repro: F.when(c, "big") pickled, then .when(c2, "p") raises PySparkTypeError
[INVALID_WHEN_USAGE]; official chains it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
