[
https://issues.apache.org/jira/browse/SPARK-61189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61189:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the metadata edited in place of a StructType in the schemas of
> mapInPandas, applyInPandas and the other map functions, in the Python client
> as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61189
> URL: https://issues.apache.org/jira/browse/SPARK-61189
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The schema of mapInPandas, mapInArrow, applyInPandas (and the state schema of
> transformWithState) was taken from the base snapshot of a StructType, which
> only add() refreshes. A metadata edit in place after the struct was built,
> st["a"].metadata["comment"] = "the a", was dropped, so the result schema had
> {} where PySpark has {'comment': 'the a'}. The schema is now converted from
> the live fields, as the other schema arguments already are. Repro:
> df.mapInPandas(f, st).schema["a"].metadata.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
