shyjsarah opened a new pull request, #8800: URL: https://github.com/apache/paimon/pull/8800
### Purpose Spark physically executes a `GlobalLimit` through a single partition. When that limited plan is used as the left input of a lateral `vector_search`, all query vectors are therefore processed by one task unless users add a repartition manually. This change: - detects a `GlobalLimit` in the lateral vector-search input; - inserts a shuffle repartition above the limit when no repartition already exists above it; - preserves an explicit user repartition placed after the limit; - adds `spark.paimon.vector-search.lateral-join.parallelism`, defaulting to 16, to control the generated parallelism. A repartition below the limit does not help because the global limit gathers its result into one partition, so the rule adds a repartition above that shape. ### Tests Added plan tests covering: - default repartition for a limited input; - configured parallelism; - repartition below a global limit; - preservation of an explicit repartition above the limit. All four added plan tests pass locally. Apache CI will run the complete Spark test matrix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
