QuakeWang opened a new pull request, #8501:
URL: https://github.com/apache/paimon/pull/8501
### Purpose
Spark create global index procedure already passes merged table/procedure
options into the topology builder, but the generic Spark builder still read
`global-index.row-count-per-shard` from table options only and used shard task
count as Spark parallelism. Procedure build options therefore could not control
non-sorted global index shard size or max Spark parallelism.
This PR reads shard size and `global-index.build.max-parallelism` from the
merged options and caps Spark parallelism with `min(taskCount,
maxParallelism)`, aligning Spark with the documented behavior and Flink generic
build semantics.
### Tests
- `mvn -pl paimon-spark/paimon-spark-common -Pspark3 -DskipTests
spotless:check`
- `mvn -pl paimon-spark/paimon-spark-common -Pspark3 -DskipTests
apache-rat:check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
