[
https://issues.apache.org/jira/browse/SPARK-61190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61190:
-----------------------------------
Labels: pull-request-available (was: )
> Keep the CASE WHEN of a pickled when() Column, so when and otherwise chain on
> it, in the Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61190
> URL: https://issues.apache.org/jira/browse/SPARK-61190
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> A when() Column that is pickled, cloudpickled or deep-copied came back as an
> opaque expression (SPARK-61167), so .when() on it raised INVALID_WHEN_USAGE
> and .otherwise() raised "otherwise() can only be applied on a Column
> previously generated by when()". PySpark unpickles a CaseWhen, so both chain.
> The same happened for a when() Column that a foreachBatch function captured,
> on the Python worker of the server.
> Repro: F.when(c, "big") pickled, then .when(c2, "p") raises PySparkTypeError
> [INVALID_WHEN_USAGE]; official chains it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
