shyjsarah commented on PR #8879: URL: https://github.com/apache/paimon/pull/8879#issuecomment-5099632557
Added a follow-up commit to balance the distributed result-merge stage. In the 100k-query / multi-partition validation, the merge stage had 16 tasks but several tasks received almost no work. The default hash partitioning of `QueryBatchId` can collide for the first batches from different input partitions. This change introduces a dedicated partitioner based on `(input partition + batch ordinal) % merge parallelism`. It keeps every identical query batch on the same reducer while spreading batches deterministically. The logic is independent of the table, filter, partition count, and configured parallelism. Validation: - `LateralVectorSearchExecutionTest`: passed on Scala 2.12 and 2.13 - `TableValuedFunctionsTest`: 23/23 passed - Spotless and diff checks passed Next I will validate the packaged change with the multi-partition workload. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
