mattcasters opened a new pull request, #7999: URL: https://github.com/apache/hop/pull/7999
Fixes #7992 This is a first version of load-balancing pipeline and workflow **run configurations**, not a new transform. Each execution is assigned to one Hop Server from a configured group. To fan out batches, use Pipeline Executor or Workflow Executor with this run configuration. ### What it does - Table of Hop Servers (name, enabled, max concurrent) in the run configuration editor - Even-load vs pack (keep as few servers busy as possible) selection - Live status probe (load, occupying unfinished pipelines/workflows, graceful shutdown) - Server-side `max_concurrent` admission (`SERVER_AT_CAPACITY`) so the cap is transactional - Client retry/wait when the pool is full or a submit is refused - Re-reads the run configuration so enable/disable and caps can change without restarting the dispatcher - Optional assignment state files plus execution-info details - Extends the existing remote engines rather than replacing them ### Also in this PR - Pipeline Executor and Workflow Executor run-configuration widgets are now `MetaSelectionLine` - Get Server Status can output finished pipeline/workflow counts - Integration test project `load-balance` with two long-lived Hop Servers: 100 child pipelines and 100 child workflows, then a post-run status check ### Docs - xref pages for the native load-balancing pipeline and workflow engines - REST `max_concurrent` on register pipeline/workflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
