dosubot[bot] commented on issue #8501: URL: https://github.com/apache/incubator-devlake/issues/8501#issuecomment-3067180735
<!-- Answer --> You're right—extract and convert steps in the Bitbucket Server plugin are slower because they process the entire dataset each run, lacking incremental or stateful logic. This is a known bottleneck and isn't addressed in current issues or pull requests. The code in `backend/plugins/bitbucket_server/tasks/` doesn't track which records have already been processed, so every run reprocesses all data, leading to inefficient SQL queries. A similar problem was solved in the GitLab plugin by restructuring pipeline stages to avoid redundant processing and reduce database growth, which improved performance significantly [(see details)](https://github.com/apache/incubator-devlake/issues/8283). For Bitbucket Server, introducing incremental extraction and conversion—such as tracking the last processed timestamp or record ID, batching, and filtering queries—would help. General MySQL optimizations like enabling slow query logging and adding indexes (e.g., on foreign keys) can also help with large datasets [(see tips)](https://github.com/apache/incubator-devlake/issues/8260). There isn't a ready-made solution for Bitbucket Server yet, but adapting the incremental/stateful processing patterns from more mature plugins is likely the best path forward. If you want to try implementing this, I can help point to specific code areas or strategies. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7bb3ffb0-f4da-46a1-bbad-9e819cffed0c?feedback_type=other)</sup> [](https://go.dosu.dev/discord-bot) [! [Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8501) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
