jrebmann opened a new pull request, #11727: URL: https://github.com/apache/nifi/pull/11727
<!-- Licensed to the Apache Software Foundation (ASF) under one or more --> <!-- contributor license agreements. See the NOTICE file distributed with --> <!-- this work for additional information regarding copyright ownership. --> <!-- The ASF licenses this file to You under the Apache License, Version 2.0 --> <!-- (the "License"); you may not use this file except in compliance with --> <!-- the License. You may obtain a copy of the License at --> <!-- http://www.apache.org/licenses/LICENSE-2.0 --> <!-- Unless required by applicable law or agreed to in writing, software --> <!-- distributed under the License is distributed on an "AS IS" BASIS, --> <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --> <!-- See the License for the specific language governing permissions and --> <!-- limitations under the License. --> # Summary [NIFI-16391](https://issues.apache.org/jira/browse/NIFI-16391) - GitLabRepositoryClient: limit commit listing to the first page instead of paging the full history GitLabRepositoryClient.getCommits(path, branch) uses the gitlab4j List overload, which internally delegates to Pager.all(). As a result, every call pages the entire commit history for the path (the client sets per_page=100). The commit listing should be bounded to the first page using a small, fixed page size (COMMIT_PAGE_SIZE). ## Environment / Observation Self-hosted GitLab, many versioned process groups. High volume of GitLab REST API calls (repository/commits) originating from NiFi. Each individual getCommits(...) call is itself several page requests, because the full history is paged through. Root Cause The use of Pager.all() in the GitLab client causes full-history paging on every call. The complete history is not required to determine the latest version or the version listing; the relevant commits are on the first page. ## Proposed Solution 1. In GitLabRepositoryClient.getCommits(path, branch), replace the Pager.all()-based overload with a call bounded to the first page. 2. Introduce a COMMIT_PAGE_SIZE constant with a small value (hard-coded initially; promotion to a property is an optional follow-up). 3. This bounds both the number of page requests per call and the payload size. ## Notes / Relationship - Mirrors the listing limit from [NIFI-14837](https://issues.apache.org/jira/browse/NIFI-14837) / PR #10186, which was applied only to GitHubRepositoryClient; GitLab does not have this limit yet. - Out of scope: This issue covers only the bounding of the commit listing in the GitLab client. Removing the per-process-group multiplier (TTL cache in AbstractGitFlowRegistryClient) and the optional (commitSha, path) content cache are not part of this ticket. - The SHA→commit-detail cache from PR #10186 is not relevant for GitLab, because the gitlab4j commit listing returns fully-populated Commit objects. References [NIFI-14837](https://issues.apache.org/jira/browse/NIFI-14837), PR https://github.com/apache/nifi/pull/10186 # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - Pull Request based on current revision of the `main` branch - Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `./mvnw clean install -P contrib-check` - [X] JDK 21 - [X] JDK 25 ### Licensing - [X] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [X] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [X] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
