JingsongLi commented on PR #8800: URL: https://github.com/apache/paimon/pull/8800#issuecomment-5045052795
**Comment 1 — generated documentation is missing** > The new `VECTOR_SEARCH_LATERAL_JOIN_PARALLELISM` option is missing from `docs/generated/spark_connector_configuration.html`, which causes `ConfigOptionsDocsCompletenessITCase` to fail in two CI jobs. Please regenerate and commit the option documentation with `mvn package -Pgenerate-docs -pl paimon-docs -nsu -DskipTests -am`, after installing the changed jars as noted in `paimon-docs/README.md`. **Comment 2 — broadcast join bypasses the repartition** > `hasUnrepartitionedGlobalLimit` stops at every non-unary node, so a broadcast join above the limited subquery bypasses this optimization even though the streamed side, and therefore the join output, still has one partition. I reproduced this with a limited left side and a broadcast right side: the physical plan contains `GlobalLimit -> Exchange SinglePartition -> BroadcastHashJoin`, no generated round-robin exchange, and `LateralVectorSearchExec` receives an RDD with one partition. Could we handle partition-preserving binary shapes such as broadcast joins, or make this decision where the effective child partitioning is available? **Comment 3 — COALESCE is incorrectly considered sufficient** > This treats every `RepartitionOperation` as sufficient, including a non-shuffle `COALESCE`. A `COALESCE(16)` above a `GlobalLimit` cannot increase its single-partition input: the physical plan remains `Coalesce 16 -> GlobalLimit -> Exchange SinglePartition`, and I verified that `LateralVectorSearchExec` still receives an RDD with one partition. Please only stop at an operation that can actually restore parallelism, such as a shuffle repartition, or explicitly continue through non-shuffle repartitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
