GitHub user dosubot[bot] added a comment to the discussion: Dora processing is too slow to complete in a few hours
<!-- Answer --> There isn't a direct flag or config to enable parallel or bulk DORA metrics processing in DevLake. The main architectural bottleneck is that tasks are processed sequentially by a single runner, so true parallelism for DORA subtasks isn't natively supported. However, you can try a few things to improve performance: - Increase the global worker concurrency by setting the `PIPELINE_MAX_PARALLEL` environment variable, which allows more pipelines to run in parallel (though this won't parallelize subtasks within a single pipeline) - Enable slow query logging in your database and add indexes to the tables used in DORA calculationsāusers have reported dramatic speedups (from hours to minutes) after optimizing indexes and queries for DORA tables like `pull_requests_*` and related foreign keys [(see discussion)](https://github.com/apache/incubator-devlake/issues/8260) - Make sure your hardware (CPU, memory) and database resources are sufficient; increasing memory has resolved similar issues for others [(see example)](https://github.com/apache/incubator-devlake/issues/8480) - Disable external monitoring tools like Dynatrace if you use them, as they've been shown to cause severe slowdowns [(see example)](https://github.com/apache/incubator-devlake/issues/8480) If you want help finding specific batch size or concurrency settings in the code, let me know! <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=hallucination) | [Report š](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/cbc26d70-60e3-4535-ad62-f7a3af58fb34?feedback_type=other)</sup> [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/discussions/8507) GitHub link: https://github.com/apache/incubator-devlake/discussions/8507#discussioncomment-13849036 ---- This is an automatically sent email for dev@devlake.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@devlake.apache.org
