klesh commented on issue #9150: URL: https://github.com/apache/devlake/issues/9150#issuecomment-5761453586
> [@klesh](https://github.com/klesh) hey as i am working in the github plugin side, so that would look into this func (c *BatchSave) Add(slot interface{}) errors.Error { if c.lastErr != nil { return errors.Default.Wrap(c.lastErr, "add slot failed due to previous err") } ... } > > func (c *BatchSave) flushWithoutLocking() errors.Error { ... err := c.db.CreateOrUpdate(c.slots.Slice(0, c.current).Interface(), clauses...) if err != nil { c.lastErr = err // <-- sticky: never cleared return err } ... } Once one batch (up to 500 rows) fails to write, lastErr is set permanently. Every later Add() in that run is rejected too — even for rows that have nothing to do with the original failure 2. Incremental cursor for performance backend/plugins/github/tasks/pr_convertor.go if stateManager.IsIncremental() { since := stateManager.GetSince() if since != nil { clauses = append(clauses, dal.Where("github_updated_at >= ?", since)) } } Future runs only re-check PRs whose GitHub updated_at changed since the last successful run. Necessary at scale — but means a PR that will never be updated again (already closed/merged) is only ever revisited once. > > Suggested fix: > > In batch_save.go: on a flush failure, retry the batch row-by-row instead of poisoning the whole run — skip/log only the row(s) that actually fail, and let the rest through. `CreateOrUpdate` is a database persistence operation. If it fails, something is seriously wrong: the entire run should stop, the code needs to be fixed, and the batch needs to be rerun to ensure data integrity. I don't see the point of salvaging those records. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
