jrt1899 opened a new pull request, #8865: URL: https://github.com/apache/texera/pull/8865
### What changes were proposed in this PR? The last of three PRs. When `network-buffering.enable-least-loaded-routing=true`, links with no partitioning requirement (`UnknownPartition`, the links that use round robin today) use `LeastLoadedPartitioning` instead of `RoundRobinPartitioning`. The flag is off by default, so existing workflows keep round robin and `RoundRobinPartitioner` is not modified. Hash, range, broadcast and one-to-one links are unaffected. > **Stacked on #8862 and #<PR2>.** Until those merge, this PR also shows their commits. Only the last commit, [`293c782`](https://github.com/jrt1899/texera/commit/293c782558bd605806bd4f3bb89b08e3a658c650), is new here. ```mermaid flowchart LR L{"Link partitioning?"} -->|"hash / range / broadcast / one-to-one"| U["unchanged"] L -->|"no requirement (UnknownPartition)"| F{"enable-least-loaded-routing"} F -->|"false (default)"| RR["RoundRobinPartitioning"] F -->|"true"| LL["LeastLoadedPartitioning"] ``` **Benchmark:** pending. It will be filled in from a run of CSV scan → Python UDF → Python UDF (8 + 8 workers, DTB 5,000, CPU) with the flag off and on. ### Any related issues, documentation, discussions? Closes: #8858 Depends on: #8862, #<PR2> Discussion: #8519 ### How was this PR tested? - `sbt "testOnly *BatchSizePropagationSpec"`: the configured batch size reaches `LeastLoadedPartitioning`. - End to end (pending): the workflow above with the flag on and off, checking identical output rows and comparing wall time. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Sonnet 5.5, Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
