[
https://issues.apache.org/jira/browse/SPARK-52450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-52450:
----------------------------------
Affects Version/s: (was: 4.1.0)
(was: 4.0.1)
> Improve performance of schema deepcopy
> --------------------------------------
>
> Key: SPARK-52450
> URL: https://issues.apache.org/jira/browse/SPARK-52450
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.0.0
> Reporter: Xi Lyu
> Assignee: Xi Lyu
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0, 4.0.1
>
>
> In Spark Connect, `DataFrame.schema` returns a deep copy of the schema to
> prevent unexpected behavior caused by user modifications to the returned
> schema object. However, if a user accesses `df.schema` repeatedly on a
> DataFrame with a complex schema, it can lead to noticeable performance
> degradation.
> Since `deepcopy` is already known for its performance overhead, we should
> consider using an alternative serialization/deserialization (ser/de) method
> to improve the efficiency of accessing `df.schema`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
