CarlosJuncher03 commented on issue #7991: URL: https://github.com/apache/hop/issues/7991#issuecomment-5319031216
Thank you. I agree that starting another JVM or container on the same machine would not create additional capacity and, without resource limits, could make contention worse. Pipeline unit tests are the appropriate solution for validating logic with small datasets. My concern is a different stage: sometimes we need to run a pipeline with a representative data volume to evaluate throughput, memory consumption, database batching, data skew, and other bottlenecks that may not appear in a unit test. In those cases, executing inside the Hop GUI JVM can make the interface unresponsive, preventing the user from monitoring or safely stopping the execution. A separate process would not make the pipeline faster, but it could provide failure and resource isolation. For example, its heap could be capped independently, while CPU limits could be applied through a container or the operating system, leaving enough resources for the GUI. If creating another engine would only reproduce the Remote Pipeline Engine, perhaps the request could be narrowed to a GUI-managed local remote runtime: start a local Hop Server process on demand, automatically provide the active project and environment, use the existing remote engine for execution and monitoring, and clean up the process afterward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
