Thundercloud12 commented on issue #9150:
URL: https://github.com/apache/devlake/issues/9150#issuecomment-5730211126
@klesh hey as i am working in the github plugin side, so that would look
into this
func (c *BatchSave) Add(slot interface{}) errors.Error {
if c.lastErr != nil {
return errors.Default.Wrap(c.lastErr, "add slot failed due to
previous err")
}
...
}
func (c *BatchSave) flushWithoutLocking() errors.Error {
...
err := c.db.CreateOrUpdate(c.slots.Slice(0, c.current).Interface(),
clauses...)
if err != nil {
c.lastErr = err // <-- sticky: never cleared
return err
}
...
}
Once one batch (up to 500 rows) fails to write, lastErr is set permanently.
Every later Add() in that run is rejected too — even for rows that have
nothing to do with the original failure
2. Incremental cursor for performance
backend/plugins/github/tasks/pr_convertor.go
if stateManager.IsIncremental() {
since := stateManager.GetSince()
if since != nil {
clauses = append(clauses, dal.Where("github_updated_at >= ?", since))
}
}
Future runs only re-check PRs whose GitHub updated_at changed since the
last successful run. Necessary at scale — but means a PR that will never be
updated again (already closed/merged) is only ever revisited once.
Suggested fix:
In batch_save.go: on a flush failure, retry the batch row-by-row instead
of poisoning the whole run — skip/log only the row(s) that actually fail,
and let the rest through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
