[
https://issues.apache.org/jira/browse/SPARK-60976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60976:
------------------------------------
Assignee: Hyukjin Kwon
> Build applyInPandasWithState as an ApplyInPandasWithState relation in Spark
> Connect Rust as PySpark does
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60976
> URL: https://issues.apache.org/jira/browse/SPARK-60976
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> {{GroupedData.applyInPandasWithState}} in the Rust-backed Python client, and
> {{GroupedData::apply_in_pandas_with_state}} in the Rust client, build a
> {{GroupMap}} relation, which the server rejects for a pandas UDF with state:
> {noformat}
> [CONNECT_INVALID_PLAN.FUNCTION_EVAL_TYPE_NOT_SUPPORTED] The Spark Connect
> plan is invalid. Function with EvalType: 208 is not supported
> {noformat}
> PySpark builds an {{ApplyInPandasWithState}} relation, with the output and
> state schemas as JSON. So every streaming query of {{applyInPandasWithState}}
> fails in the client, while it works in PySpark 4.3.0. Build the relation as
> PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
