Hyukjin Kwon created SPARK-61207:
------------------------------------
Summary: StructType.add re-reads every field and its metadata in
the Python client
Key: SPARK-61207
URL: https://issues.apache.org/jira/browse/SPARK-61207
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Since SPARK-61178, StructType.add re-reads every field of the struct on each
call. That includes the JSON encoding of every field's metadata and, for a
field of a struct, array or map type, every nested field. In the Python client,
a loop of 1000 add(name, StringType(), True, {"comment": ...}) calls takes
about 1.6 s, against 55 ms with official PySpark 4.3.0. 1000 adds of commented
struct, array and map types take about 4 s, against 0.1 s. PySpark's add
appends the new field and computes the needConversion of each field from its
type. With the fix, add reads the metadata of the new field only and keeps the
needConversion of the fields of atomic types. It reads that of nested types
from the type objects, without reading their fields.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
