caioq commented on issue #7826: URL: https://github.com/apache/incubator-devlake/issues/7826#issuecomment-2821946978
Hi team! @klesh @wouldd I’ve been returned to investigate this issue and would like to request that this issue be reopened so we can collaborate on a proper fix. **Here’s what I’ve found:** - I noticed that errors like `unexpected EOF` and `busy buffer` are being raised during the `extractIssues` step. The problem is that these errors don’t appear in the downloadable pipeline logs, only in the application logs.  - When this happens, the pipeline step still finishes successfully, but any Issues that weren’t extracted from `_raw_jira_api_issues` to `_tool_jira_issues` are silently lost until a completed successful re-run — which means we’re dealing with data loss on partial extraction failures. - I was able to reliably reproduce this issue locally by reducing some MySQL database parameters: ```yaml --max_allowed_packet=9M --net_read_timeout=6 --net_write_timeout=6 ``` This made it easier to trigger the `unexpected EOF` and `busy buffer` errors during extractions. Example of mysql container in docker-compose: ```yaml mysql: image: mysql:8 volumes: - mysql-storage:/var/lib/mysql restart: always ports: - 3306:3306 environment: MYSQL_ROOT_PASSWORD: admin MYSQL_DATABASE: lake MYSQL_USER: merico MYSQL_PASSWORD: merico TZ: UTC command: --character-set-server=utf8mb4 --collation-server=utf8mb4_bin --skip-log-bin --max_allowed_packet=9M --net_read_timeout=6 --net_write_timeout=6 ``` - I attempted to fix this by reviewing all error handling in the `Add` (https://github.com/apache/incubator-devlake/blob/main/backend/helpers/pluginhelper/api/batch_save.go#L81) and `flushWithoutLocking`(https://github.com/apache/incubator-devlake/blob/main/backend/helpers/pluginhelper/api/batch_save.go#L123) methods to make sure errors propagate correctly, but I wasn’t able to catch this particular error flow. Would you have any suggestions on how to properly detect and surface these errors so the pipeline step can fail when this happens, instead of silently proceeding? @wouldd — do you think this could be related to the issue you experienced? I also believe this might be affecting other extraction processes, as seen in: https://github.com/apache/incubator-devlake/issues/8198 https://github.com/apache/incubator-devlake/issues/8349 This problem is still frequently impacting our use of DevLake in our production environment, and I’d sincerely appreciate the team’s help in addressing this. Thanks so much in advance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
