vbhanuchander-lang opened a new pull request, #9042:
URL: https://github.com/apache/devlake/pull/9042
Closes #8834
### The mechanism is the incremental filter, not the LEFT JOIN
The thread converged on the `LEFT JOIN` + `WHERE board_id` anti-pattern, and
@klesh reasonably rejected the proposed fix because moving `board_id` into the
`ON` clause makes every board task convert the whole connection. I think the
actual defect is somewhere else, and the fix costs nothing in scope or
performance.
The convertor's incremental filter is:
```go
dal.Where("_tool_jira_issue_changelog_items.created_at >= ? ", since)
```
`created_at` comes from `common.NoPKModel` and is stamped when the row is
**first inserted**. It never moves again. So a changelog item collected during
one window but not converted in that window can never be selected by any later
incremental run — the column it is filtered on is permanently in the past.
Nothing errors. The rows just stay in the tool layer.
That fits every symptom in the report:
| Reported | Explained by |
|---|---|
| Partial, ~79% converted / ~21% not, same sync run | window boundary, not
board membership |
| Persists across runs | `created_at` cannot move, so the item is
permanently excluded |
| No errors, no warnings | nothing failed — the rows were never selected |
| 4%–99% across projects | depends on how much of each project's history
predates the window |
It also explains why the "not the incremental filter" check came out clean:
`_devlake_collector_latest_state` is the **collector's** state. The convertor
keeps its own subtask state, and that is what gates this query.
### The fix
```diff
-dal.Where("_tool_jira_issue_changelog_items.created_at >= ? ", since)
+dal.Where("_tool_jira_issue_changelog_items.updated_at >= ? ", since)
```
`updated_at` is refreshed by the extractor's upsert (`CreateOrUpdate` →
`OnConflict{UpdateAll: true}`), so a re-collected item is reconsidered and a
re-collection actually repairs the gap.
Consistency check across the code base — of 36 convertors, this was **the
only one** filtering on `created_at`:
```
$ grep -rh "dal.Where(\"" --include="*convertor*.go" plugins/ | grep -E
"(created|updated)_at >= "
35 updated_at
1 created_at <- plugins/jira/tasks/issue_changelog_convertor.go
```
**Crucially, this does not widen the board filter.** No board task converts
anything outside its own board, so the cost @klesh objected to does not arise.
### Making the board filter non-silent
The board filter is deliberate and I have left it alone. But the issue also
asks that the convertor not skip records silently, and that part is fair
independently of the bug above.
After a successful conversion the subtask now counts collected changelog
items whose issue is not on this board, and logs the number with the reason.
One aggregate query per board task, negligible next to the conversion. It turns
"21% of my changelogs are missing" into a logged figure with an explanation. A
failure in the diagnostic is logged, never propagated — it cannot turn a good
run into a failed one.
@ciaramulligan — for overlapping boards this means the shortfall becomes
visible and attributable rather than mysterious, and with the filter fix a
re-collection now repairs previously stranded items.
### Tests
`TestIssueChangelogBoardScopeDataFlow` (e2e, ran against MySQL 8.4) drives
the convertor with a changelog for an issue on no board and asserts: the
out-of-scope item is not converted, nothing is attached to
`jira:JiraIssue:2:0`, and in-scope changelogs still convert — so the exclusion
is scoping, not a regression that dropped everything. The diagnostic is
observably firing in that run:
```
level=warning msg="1 collected changelog item(s) are not associated with
board 8 and were not
converted; they belong to issues outside this board's scope. ..."
```
`TestIncrementalFilterUsesUpdatedAtNotCreatedAt` guards the column choice.
Reverting it is a one-token change that silently restores permanent data loss
and nothing else in the suite would catch it.
Full plugin, with a database attached:
```
ok github.com/apache/incubator-devlake/plugins/jira/api 0.221s
ok github.com/apache/incubator-devlake/plugins/jira/e2e 11.106s
ok github.com/apache/incubator-devlake/plugins/jira/tasks 0.254s
ok github.com/apache/incubator-devlake/plugins/jira/tasks/apiv2models 0.418s
```
### What I could not verify
I have no access to the reporter's Jira instance, so I have **not**
reproduced the original 21% shortfall end to end. The mechanism above is
established from the code, the model definitions and the upsert behaviour, and
it is consistent with every reported symptom — but the confirmation that would
close it beyond doubt is @ciaramulligan running a sync on this branch and
comparing tool-layer and domain-layer counts again.
If the shortfall persists after this, the remaining suspect is genuine board
non-membership, and the new warning will say exactly how many items that
accounts for — which turns the next round of diagnosis into reading a log line
instead of writing SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
