vbhanuchander-lang commented on PR #9027:
URL: https://github.com/apache/devlake/pull/9027#issuecomment-5159394878
Thanks for looking @klesh. I may have described this poorly, because there
aren't missing *fields* here — the raw record itself is gone.
The extractor cursor LEFT JOINs the raw table on `_raw_data_id`. When the
referenced raw row no longer exists, `data` comes back NULL. There is no JSON
to extract from, and nothing that could be added, because the source record was
legitimately deleted.
It happens routinely rather than exceptionally. `ApiCollector.Execute()`
deletes every raw row for the params at the start of any non-incremental
collection and re-inserts them with fresh auto-increment ids:
```go
// helpers/pluginhelper/api/api_collector.go
if !isIncremental {
err = db.Delete(&RawData{}, dal.From(collector.table), dal.Where("params
= ?", collector.params))
}
```
Any domain row still carrying an id from an earlier run is orphaned by that,
so a single full re-collect is enough to produce this state.
If you meant clearing the custom fields rather than leaving them as they are
— I did consider it, and I think it is worse: it would erase previously
extracted values on every full re-collect, turning a routine operation into
data loss. Leaving them and reporting the count seemed the safer default.
If you had something else in mind — for example re-linking domain rows to
the new raw ids so the data can be re-extracted — I am happy to look at that,
though it would belong in the collector rather than the customize plugin. Which
would you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
