CarlosJuncher03 commented on issue #7992: URL: https://github.com/apache/hop/issues/7992#issuecomment-5330196115
Right, @mattcasters . I agree with you—and with @hansva —that it is possible to handle this type of scenario using an orchestrator like Airflow. I also understand that, as it stands, one can already build logic directly within Hop to decide which Hop Server should run a specific pipeline, utilizing another pipeline, variables, and existing components. The idea I was trying to propose is precisely the one you mentioned: a "Load Balancing Pipeline Run Configuration" that includes multiple Hop Servers. The only additional point I’d like to clarify concerns the behavior when a parent pipeline triggers multiple executions of the same child pipeline—for instance, by passing different values from the parent pipeline as parameters. Something like this: Parent Pipeline | +-- value A --> Child Pipeline --> Hop Server 1 +-- value B --> Child Pipeline --> Hop Server 2 +-- value C --> Child Pipeline --> Hop Server 3 +-- value D --> Child Pipeline --> next available Hop Server In this case, I would like the Load Balancing Run Configuration to treat each execution of the child pipeline as an independent run and automatically select an available Hop Server from the pool. I am not proposing splitting transforms or distributing the internal rows of a single pipeline execution across multiple servers. Each Hop Server would still execute a complete instance of the child pipeline; the load balancing would occur only across the different executions of that pipeline. That is basically the scenario I was trying to describe. And again, sorry if I’m not explaining the idea correctly; sometimes the translator changes terms and even what I’m trying to write. It seems Google Translate now has built-in AI that changes words based on how it understands the context. Anyway, thanks for everything—I think these exchanges of ideas are amazing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
