zhuxiangyi opened a new pull request, #8943:
URL: https://github.com/apache/paimon/pull/8943
### What is the purpose of the change
This PR optimizes `SparkInternalRow.create(rowType)` for very wide nested
`RowType`s.
When Spark reads a Paimon table containing a wide struct, for example a
struct with tens of thousands of fields, nested struct conversion can
repeatedly create `SparkInternalRow` wrappers. Before this change, each
creation scanned all fields in the `RowType` to find BLOB fields.
In production, executor threads can spend a significant amount of time in
the following stack:
```text
org.apache.paimon.types.RowType.getFieldCount
org.apache.paimon.spark.data.SparkInternalRow$.blobFields
org.apache.paimon.spark.data.SparkInternalRow$.create
org.apache.paimon.spark.DataConverter.fromPaimon
org.apache.paimon.spark.AbstractSparkInternalRow.getStruct
```
This repeated scan is expensive for very wide structs.
### Brief change log
- Cache BLOB field lookup results in `SparkInternalRow`.
- Use `IdentityHashMap[RowType, Set[Int]]` so cache lookup does not call
`RowType.hashCode` or `RowType.equals`, which may also traverse wide schemas.
- Add a regression test to verify that repeated
`SparkInternalRow.create(rowType)` calls for the same `RowType` scan the fields
only once.
### Verifying this change
This change added tests and can be verified as follows:
```bash
mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 \
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.data.SparkInternalRowTest \
-Dtest=none test
```
Result:
```text
BUILD SUCCESS
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
