Jannik Rebmann created NIFI-16391:
-------------------------------------
Summary: GitLabRepositoryClient: limit commit listing to the first
page instead of paging the full history
Key: NIFI-16391
URL: https://issues.apache.org/jira/browse/NIFI-16391
Project: Apache NiFi
Issue Type: Improvement
Components: Flow Versioning
Affects Versions: 2.12.0
Reporter: Jannik Rebmann
h4. Summary
{{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}}
overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every
call pages the *entire commit history* for the path (the client sets
{{{}per_page=100{}}}). The commit listing should be bounded to the first page
using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
h4. Environment / Observation
* Self-hosted GitLab, many versioned process groups.
* High volume of GitLab REST API calls ({{{}repository/commits{}}})
originating from NiFi.
* Each individual {{getCommits(...)}} call is itself several page requests,
because the full history is paged through.
h4. Root Cause
The use of {{Pager.all()}} in the GitLab client causes full-history paging on
every call. The complete history is not required to determine the latest
version or the version listing; the relevant commits are on the first page.
h4. Proposed Solution
# In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the
{{{}Pager.all(){}}}-based overload with a call bounded to the first page.
# Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded
initially; promotion to a property is an optional follow-up).
# This bounds both the number of page requests per call and the payload size.
h4. Notes / Relationship
* Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was
applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit
yet.
* *Out of scope:* This issue covers only the bounding of the commit listing in
the GitLab client. Removing the per-process-group multiplier (TTL cache in
{{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}}
content cache are *not* part of this ticket.
* The SHA→commit-detail cache from PR #10186 is not relevant for GitLab,
because the gitlab4j commit listing returns fully-populated {{Commit}} objects.
h4. References
NIFI-14837, PR [https://github.com/apache/nifi/pull/10186]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
