wouldd commented on issue #7826: URL: https://github.com/apache/incubator-devlake/issues/7826#issuecomment-2331988108
@klesh so I think I may have finally tripped this scenario whilst slowly controlling the refreshes but also putting hte system under some load. I n this case the jira refresh starts, the database wipes the info, but then the collector fails because jira throws a 429 - too many requests, I'll paste the full error at the end. I would expect in this scenario that it would not wipe out the database until the collector has been successful. By default pipelines skip failed tasks, which makes it hard to see when there has been a failure since the pipeline often just shows 'partial success' and if I'm looking after the daily node recycle then I can't get the logs. once this has happend the next 'normal' refresh would only collect issues that have updated since the last pipeline even though it essentially failed this critical step. Consequently once the data is gone, it never comes back without a full refresh running successfully does that sound about right? basically any error in the collector during the pipeline leaves you with no data even if you had it before, and only a successful full refresh can restore it. That feels to me like a problem with sequencing and error handling? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
