shashank created CAMEL-25807:
--------------------------------
Summary: camel-github2 - the commit consumer silently skips the
older commits when more than 100 commits are pushed between two polls
Key: CAMEL-25807
URL: https://issues.apache.org/jira/browse/CAMEL-25807
Project: Camel
Issue Type: Bug
Components: camel-github
Reporter: shashank
Assignee: shashank
{{CommitConsumer.poll}} (main 7aae27315946, lines 104-165) lists the commits of
the branch, newest first, and stops after 100 ("Stop after 100 commits to avoid
fetching too many", lines 113-117). It then clips the list at the last routed
commit ({{lastSha}}) and routes the commits before it, oldest first;
{{onRouting}} moves {{lastSha}} to each routed commit, so it ends at the newest
one.
When more than 100 commits were pushed to the branch between two polls,
{{lastSha}} is not among the 100 newest: the poll routes those 100 (the ones
not in the remembered set), {{lastSha}} becomes the newest commit, and the next
poll stops at it. The older new commits are never routed, and nothing is
logged. This happens with a large merge or push, and at the first poll after
GitHub was unreachable (or the polls failed, for example on the rate limit)
while more than 100 commits were pushed.
*Fix*: when there is a last routed commit, the poll reads the listing up to it,
at most 1000 commits (the number of commits the consumer already remembers in
its set of routed shas). When the last commit is not found (it is no longer on
the branch, after a force push) or there is none ({{startingSha=beginning}}),
only the 100 newest commits are considered, as before. Upgrade note in the 4.24
guide.
*Tests*: new {{GitHub2CommitConsumerManyNewCommitsTest}} (mocked repository
listing, as in {{GitHub2ConsumerPositionTest}}). {{everyNewCommitIsRouted}}
(150 commits pushed after {{startingSha=c0}}) fails on main: the routed list is
{{[c51, c52, ..., c150]}} instead of {{[c1, ..., c150]}}; passes with the fix.
{{withoutTheLastCommitOnTheBranchOnlyTheNewest100AreRouted}} passes on main and
with the fix (the force push case keeps its behaviour). camel-github2 unit
tests: 16 run, 0 failures.
*How it was found*: the same Z3 model as the camel-jira newIssues consumer (a
cursor that moves to the newest item of a poll whose answer is cut to the
newest M): an item is lost as soon as M+1 items arrive between two polls, none
when the poll reads up to the cursor; then the JUnit test above. A variant of
CAMEL-25806 (camel-jira newIssues).
Related: CAMEL-21256 ({{startingSha=last}} with a large history). When there is
no {{lastSha}}, or it is not in the listing (a force push), the poll still
reads at most the newest 100 commits as before, so that case is unchanged.
*Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same cut at
100; read, not run). camel-4.14.x has no camel-github2.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
