[
https://issues.apache.org/jira/browse/NIFI-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jannik Rebmann updated NIFI-16391:
----------------------------------
Description:
h4. Summary
{{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}}
overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every
call pages the *entire commit history* for the path (the client sets
{{{}per_page=100{}}}). The commit listing should be bounded to the first page
using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
h4. Environment / Observation
* Self-hosted GitLab, many versioned process groups.
* High volume of GitLab REST API calls ({{{}repository/commits{}}})
originating from NiFi.
* Each individual {{getCommits(...)}} call is itself several page requests,
because the full history is paged through.
h4. Root Cause
The use of {{Pager.all()}} in the GitLab client causes full-history paging on
every call. The complete history is not required to determine the latest
version or the version listing; the relevant commits are on the first page.
h4. Proposed Solution
# In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the
{{{}Pager.all(){}}}-based overload with a call bounded to the first page.
# Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded
initially; promotion to a property is an optional follow-up).
# This bounds both the number of page requests per call and the payload size.
h4. Notes / Relationship
* Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was
applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit
yet.
* *Out of scope:* This issue covers only the bounding of the commit listing in
the GitLab client. Removing the per-process-group multiplier (TTL cache in
{{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}}
content cache are *not* part of this ticket.
* The SHA→commit-detail cache from PR #10186 is not relevant for GitLab,
because the gitlab4j commit listing returns fully-populated {{Commit}} objects.
h4. References
NIFI-14837, NIFI-16359, PR [https://github.com/apache/nifi/pull/10186]
was:
h4. Summary
{{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}}
overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every
call pages the *entire commit history* for the path (the client sets
{{{}per_page=100{}}}). The commit listing should be bounded to the first page
using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
h4. Environment / Observation
* Self-hosted GitLab, many versioned process groups.
* High volume of GitLab REST API calls ({{{}repository/commits{}}})
originating from NiFi.
* Each individual {{getCommits(...)}} call is itself several page requests,
because the full history is paged through.
h4. Root Cause
The use of {{Pager.all()}} in the GitLab client causes full-history paging on
every call. The complete history is not required to determine the latest
version or the version listing; the relevant commits are on the first page.
h4. Proposed Solution
# In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the
{{{}Pager.all(){}}}-based overload with a call bounded to the first page.
# Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded
initially; promotion to a property is an optional follow-up).
# This bounds both the number of page requests per call and the payload size.
h4. Notes / Relationship
* Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was
applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit
yet.
* *Out of scope:* This issue covers only the bounding of the commit listing in
the GitLab client. Removing the per-process-group multiplier (TTL cache in
{{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}}
content cache are *not* part of this ticket.
* The SHA→commit-detail cache from PR #10186 is not relevant for GitLab,
because the gitlab4j commit listing returns fully-populated {{Commit}} objects.
h4. References
NIFI-14837, PR [https://github.com/apache/nifi/pull/10186]
> GitLabRepositoryClient: limit commit listing to the first page instead of
> paging the full history
> -------------------------------------------------------------------------------------------------
>
> Key: NIFI-16391
> URL: https://issues.apache.org/jira/browse/NIFI-16391
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Flow Versioning
> Affects Versions: 2.12.0
> Reporter: Jannik Rebmann
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h4. Summary
> {{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j
> {{List}} overload, which internally delegates to {{{}Pager.all(){}}}. As a
> result, every call pages the *entire commit history* for the path (the client
> sets {{{}per_page=100{}}}). The commit listing should be bounded to the first
> page using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
> h4. Environment / Observation
> * Self-hosted GitLab, many versioned process groups.
> * High volume of GitLab REST API calls ({{{}repository/commits{}}})
> originating from NiFi.
> * Each individual {{getCommits(...)}} call is itself several page requests,
> because the full history is paged through.
> h4. Root Cause
> The use of {{Pager.all()}} in the GitLab client causes full-history paging on
> every call. The complete history is not required to determine the latest
> version or the version listing; the relevant commits are on the first page.
> h4. Proposed Solution
> # In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the
> {{{}Pager.all(){}}}-based overload with a call bounded to the first page.
> # Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded
> initially; promotion to a property is an optional follow-up).
> # This bounds both the number of page requests per call and the payload size.
> h4. Notes / Relationship
> * Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was
> applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this
> limit yet.
> * *Out of scope:* This issue covers only the bounding of the commit listing
> in the GitLab client. Removing the per-process-group multiplier (TTL cache in
> {{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}}
> content cache are *not* part of this ticket.
> * The SHA→commit-detail cache from PR #10186 is not relevant for GitLab,
> because the gitlab4j commit listing returns fully-populated {{Commit}}
> objects.
> h4. References
> NIFI-14837, NIFI-16359, PR [https://github.com/apache/nifi/pull/10186]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
