[
https://issues.apache.org/jira/browse/SPARK-61207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61207.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 393
[https://github.com/apache/spark-connect-rust/pull/393]
> StructType.add re-reads every field and its metadata in the Python client
> -------------------------------------------------------------------------
>
> Key: SPARK-61207
> URL: https://issues.apache.org/jira/browse/SPARK-61207
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Since SPARK-61178, StructType.add re-reads every field of the struct on each
> call. That includes the JSON encoding of every field's metadata and, for a
> field of a struct, array or map type, every nested field. In the Python
> client, a loop of 1000 add(name, StringType(), True, {"comment": ...}) calls
> takes about 1.6 s, against 55 ms with official PySpark 4.3.0. 1000 adds of
> commented struct, array and map types take about 4 s, against 0.1 s.
> PySpark's add appends the new field and computes the needConversion of each
> field from its type. With the fix, add reads the metadata of the new field
> only and keeps the needConversion of the fields of atomic types. It reads
> that of nested types from the type objects, without reading their fields.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
