dosubot[bot] commented on issue #8678: URL: https://github.com/apache/incubator-devlake/issues/8678#issuecomment-3723409680
<!-- Greeting --> Hi @dhruv-paystack! I'm [Dosu](https://go.dosu.dev/dosubot) and Iām helping the incubator-devlake team. <!-- Answer --> The error happens because the LinkPrToIssue subtask builds a SQL query with large `IN` clauses for issue IDs and keys, which can easily exceed MySQL's placeholder limit (often 500) when you have many repos or JIRA projects. The code currently does not chunk these lists, so a single query can blow past the limit and trigger the error [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/linker/tasks/link_pr_and_issue.go#L48-L141). To fix this, you can adapt the batching pattern used elsewhere in the codebase (see `BatchSaveDivider` and `DataEnricher`) to chunk your SELECT queries. Specifically, split `projectIssueIds` and/or `issueKeys` into smaller slices (e.g., batches of 400ā500), and run multiple queries instead of one massive one. This keeps each query under the placeholder limit and avoids the error [source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/data_enricher.go#L41-L121). For your PR: focus on the `LinkPrToIssue` function in `backend/plugins/linker/tasks/link_pr_and_issue.go`. Refactor the part that queries issues to loop over chunks of the IDs/keys, collect results, and proceed as usual. You can borrow chunking logic from the batch helpers, or write a simple utility to split slices. As a temporary workaround, you could reduce the number of repos or JIRA projects processed in a single pipeline run, but a code fix is the robust solution. Let me know if you want a code snippet or more detailed guidance for the chunking logic! <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=hallucination) | [Report š](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/b0fb63f9-a835-43b5-a712-3b04efe5be06?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=github) [](https://cursor.com/link/prompt?text=The%20error%20happens%20because%20the%20LinkPrToIssue%20subtask%20builds%20a%20SQL%20query%20with%20large%20%60IN%60%20clauses%20for%20issue%20IDs%20and%20keys%2C%20which%20can%20easily%20exceed%20MySQL%27s%20placeholder%20limit%20%28often%20500%29%20when%20you%20have%20many%20repos%20or%20JIRA%20projects.%20The%20code%20currently%20does%20not%20chunk%20these%20lists%2C%20so%20a%20single%20query%20can%20blow%20past%20the%20limit%20and%20trigger%20the%20error%20%5Bsource%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/linker/tasks/link_pr_and_issue.go%23L48-L141%29.%0A%0ATo%20fix%20this%2C%20you%20can%20adapt%20the%20batching%20pattern%20used%20elsewhere%20in%20the%20codebase%20%28see% 20%60BatchSaveDivider%60%20and%20%60DataEnricher%60%29%20to%20chunk%20your%20SELECT%20queries.%20Specifically%2C%20split%20%60projectIssueIds%60%20and/or%20%60issueKeys%60%20into%20smaller%20slices%20%28e.g.%2C%20batches%20of%20400%E2%80%93500%29%2C%20and%20run%20multiple%20queries%20instead%20of%20one%20massive%20one.%20This%20keeps%20each%20query%20under%20the%20placeholder%20limit%20and%20avoids%20the%20error%20%5Bsource%5D%28https%3A//github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/helpers/pluginhelper/api/data_enricher.go%23L41-L121%29.%0A%0AFor%20your%20PR%3A%20focus%20on%20the%20%60LinkPrToIssue%60%20function%20in%20%60backend/plugins/linker/tasks/link_pr_and_issue.go%60.%20Refactor%20the%20part%20that%20queries%20issues%20to%20loop%20over%20chunks%20of%20the%20IDs/keys%2C%20collect%20results%2C%20and%20proceed%20as%20usual.%20You%20can%20borrow%20chunking%20logic%20from%20the%20batch%20helpers%2C%20or%20write%20a%20simple%20utility%2 0to%20split%20slices.%0A%0AAs%20a%20temporary%20workaround%2C%20you%20could%20reduce%20the%20number%20of%20repos%20or%20JIRA%20projects%20processed%20in%20a%20single%20pipeline%20run%2C%20but%20a%20code%20fix%20is%20the%20robust%20solution.%0A%0ALet%20me%20know%20if%20you%20want%20a%20code%20snippet%20or%20more%20detailed%20guidance%20for%20the%20chunking%20logic%21) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8678) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
