renflo opened a new issue, #5852: URL: https://github.com/apache/incubator-devlake/issues/5852
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened As mentioned in https://github.com/apache/incubator-devlake/issues/5849 I have a few issues and periodical pipelines failing. My periodical jobs tend to happen at the same time and that when it happens they more or less all fail. I've seen errors about database connection not being available and the 429 bug mentioned over will of course also be very likely to occur. I can't recall if I've also had out of memory errors but it's possible. Anyways: in many case a pipeline triggered manually when no other pipeline runs succeeds, but if I look at the nightly run this exact same pipeline, running at the same time as the others, will have failed. That means that I end up having to trigger all pipelines manually at regular interval to get fresh data. That's not sustainable. ### What do you expect to happen Making sure devlake never fails because of concurrency is of course ideal, and https://github.com/apache/incubator-devlake/issues/5849 which I just filed is falling in this category of requests. But I would really like you to add a configurable `MAX_CONCURRENT_PIPELINES` configuration variable to the system which would lead to it never running more pipeline than configured in parallel. An env var, like the many other env vars, would be perfect. ### How to reproduce Create many concurrent pipeline and run them all together. ### Anything else _No response_ ### Version v0.18.0-beta4 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
