shyjsarah commented on PR #8800: URL: https://github.com/apache/paimon/pull/8800#issuecomment-5058475363
Addressed the statistics-based automatic broadcast case in commit `31bf5d0de1`. - Added a physical-plan fallback in `LateralVectorSearchExec.requiredChildDistribution`, where Spark's selected broadcast build side is available. - Followed the streamed side of `BroadcastHashJoinExec` and `BroadcastNestedLoopJoinExec` through partition-preserving unary operators. - Restored the configured lateral-search parallelism only when that streamed path still contains an unrepartitioned `GlobalLimit`. - Preserved existing streamed-side parallelism for `BuildLeft` and stopped when a shuffle already exists. - Added no-hint, statistics-based broadcast tests for both build sides with AQE enabled and disabled. - Added a version shim for `ClusteredDistribution` because Spark 3.2 uses the two-argument constructor. Validation: - `TableValuedFunctionsTest`: 22/22 passed. - Spark 3.2 and 3.3 compile with Scala 2.12 and 2.13. - Spark 4.0 compiles with JDK 17. - Verified the Spark 3.2 shaded bytecode calls the two-argument `ClusteredDistribution` constructor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
