Rarasinhoo commented on code in PR #11926:
URL: https://github.com/apache/nifi/pull/11926#discussion_r4593376884


##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,183 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
+
+    private final LongSupplier nanoTimeSupplier;
+    private final Map<String, BranchState> branchStates = new 
ConcurrentHashMap<>();
+
+    GitBranchStateCache() {
+        this(System::nanoTime);
+    }
+
+    GitBranchStateCache(final LongSupplier nanoTimeSupplier) {
+        this.nanoTimeSupplier = nanoTimeSupplier;
+    }
+
+    /**
+     * Gets the state of the branch, checking its head when the last check is 
older than the check interval.
+     * When the head moved, values cached for files changed between the two 
heads are dropped; when those files
+     * cannot be determined, every value cached for the branch is dropped.
+     *
+     * @param repositoryClient the repository client
+     * @param branch the branch
+     * @return the state of the branch, or empty when the repository client 
cannot report the branch head
+     */
+    Optional<BranchState> getState(final GitRepositoryClient repositoryClient, 
final String branch) throws IOException, FlowRegistryException {
+        final long now = nanoTimeSupplier.getAsLong();
+        final BranchState state = branchStates.get(branch);
+        if (state != null && now - state.checkedAt < 
HEAD_CHECK_INTERVAL_NANOS) {
+            return Optional.of(state);
+        }
+
+        final Optional<String> headSha = 
repositoryClient.getBranchHeadSha(branch);
+        if (headSha.isEmpty()) {
+            branchStates.remove(branch);
+            return Optional.empty();
+        }
+
+        if (state != null && state.headSha.equals(headSha.get())) {
+            state.checkedAt = now;
+            return Optional.of(state);
+        }
+
+        final BranchState currentState = new BranchState(headSha.get(), now);
+        if (state != null) {
+            final Optional<Set<String>> changedPaths = 
repositoryClient.getChangedPaths(state.headSha, currentState.headSha);

Review Comment:
   Agreed. In 9c941684 a failed comparison is treated as unknown changes: the 
cache moves to the new head with an empty state for the branch, so later calls 
read from GitHub instead of failing until a restart. 
`testFailedComparisonDropsEverythingAndMovesToNewHead` covers a Not Found from 
the comparison and checks the next call does not compare again.



##########
nifi-extension-bundles/nifi-github-bundle/nifi-github-extensions/src/main/java/org/apache/nifi/github/GitHubRepositoryClient.java:
##########
@@ -337,6 +341,75 @@ public List<GitCommit> getCommits(final String path, final 
String branch) throws
         });
     }
 
+    /**
+     * Gets the SHA of the commit at the head of the given branch with a 
single request.
+     *
+     * @param branch the branch
+     * @return the SHA of the head commit
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling 
GitHub, or the branch does not exist
+     */
+    @Override
+    public Optional<String> getBranchHeadSha(final String branch) throws 
IOException, FlowRegistryException {
+        final String branchRef = BRANCH_REF_PATTERN.formatted(branch);
+        logger.debug("Getting head commit of branch [{}] in repo [{}]", 
branch, repository.getName());
+
+        return execute(() -> {
+            try {
+                return 
Optional.of(repository.getRef(branchRef).getObject().getSha());
+            } catch (final FileNotFoundException fnf) {
+                throw new FlowRegistryException("Branch [" + branchRef + "] 
not found", fnf);
+            }
+        });
+    }
+
+    /**
+     * Gets the paths changed from one commit to a later one with a single 
compare request.
+     *
+     * @param fromCommitSha the SHA of the earlier commit
+     * @param toCommitSha the SHA of the later commit
+     * @return the changed paths relative to the repository path, or empty 
when the later commit does not descend from
+     *         the earlier one or the comparison lists as many files as GitHub 
returns at most
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling GitHub
+     */
+    @Override
+    public Optional<Set<String>> getChangedPaths(final String fromCommitSha, 
final String toCommitSha) throws IOException, FlowRegistryException {
+        logger.debug("Getting paths changed from [{}] to [{}] in repo [{}]", 
fromCommitSha, toCommitSha, repository.getName());
+        return execute(() -> 
getChangedPaths(repository.getCompare(fromCommitSha, toCommitSha), repoPath));
+    }
+
+    static Optional<Set<String>> getChangedPaths(final GHCompare compare, 
final String repoPath) {
+        final GHCompare.Status status = compare.getStatus();
+        if (status != GHCompare.Status.ahead && status != 
GHCompare.Status.identical) {
+            return Optional.empty();
+        }
+
+        final GHCommit.File[] files = compare.getFiles();
+        if (files.length >= COMPARE_FILES_LIMIT) {

Review Comment:
   Not acceptable, you are right: the cached latest version would point to an 
older commit. In 9c941684 the changed paths are the paths touched by any commit 
in the range. When one commit separates the two heads the comparison's file 
list is exactly that commit's files, so it stays one request. With more 
commits, each commit's files are read (one request per commit, at most 20); 
beyond that, or when GitHub truncates the commit list, the whole branch is 
dropped. Verified against GitHub on a range of 6 commits (1,188 touched files) 
and of 1 commit (76 files, compare only); 
`testGetChangedPathsIncludesFilesRevertedWithinRange` covers a file changed and 
reverted inside the range.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,183 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
+
+    private final LongSupplier nanoTimeSupplier;
+    private final Map<String, BranchState> branchStates = new 
ConcurrentHashMap<>();
+
+    GitBranchStateCache() {
+        this(System::nanoTime);
+    }
+
+    GitBranchStateCache(final LongSupplier nanoTimeSupplier) {
+        this.nanoTimeSupplier = nanoTimeSupplier;
+    }
+
+    /**
+     * Gets the state of the branch, checking its head when the last check is 
older than the check interval.
+     * When the head moved, values cached for files changed between the two 
heads are dropped; when those files
+     * cannot be determined, every value cached for the branch is dropped.
+     *
+     * @param repositoryClient the repository client
+     * @param branch the branch
+     * @return the state of the branch, or empty when the repository client 
cannot report the branch head
+     */
+    Optional<BranchState> getState(final GitRepositoryClient repositoryClient, 
final String branch) throws IOException, FlowRegistryException {
+        final long now = nanoTimeSupplier.getAsLong();
+        final BranchState state = branchStates.get(branch);
+        if (state != null && now - state.checkedAt < 
HEAD_CHECK_INTERVAL_NANOS) {
+            return Optional.of(state);
+        }
+
+        final Optional<String> headSha = 
repositoryClient.getBranchHeadSha(branch);
+        if (headSha.isEmpty()) {
+            branchStates.remove(branch);
+            return Optional.empty();
+        }
+
+        if (state != null && state.headSha.equals(headSha.get())) {
+            state.checkedAt = now;
+            return Optional.of(state);
+        }
+
+        final BranchState currentState = new BranchState(headSha.get(), now);
+        if (state != null) {
+            final Optional<Set<String>> changedPaths = 
repositoryClient.getChangedPaths(state.headSha, currentState.headSha);
+            changedPaths.ifPresent(paths -> 
currentState.retainUnchanged(state, paths));
+        }
+        // Values put into the replaced state by requests still running are 
dropped with it
+        branchStates.put(branch, currentState);
+        return Optional.of(currentState);
+    }
+
+    /**
+     * Drops the values cached for the given file, after this client changed 
it.
+     *
+     * @param branch the branch
+     * @param path the path of the file
+     */
+    void invalidate(final String branch, final String path) {
+        final BranchState state = branchStates.get(branch);
+        if (state != null) {
+            state.remove(path);
+        }
+    }
+
+    void clear() {
+        branchStates.clear();
+    }
+
+    /**
+     * The latest commit of a file as read from the repository; the SHA is 
null when no commit changed the file.
+     */
+    record LatestVersion(String commitSha) {
+        Optional<String> toOptional() {
+            return Optional.ofNullable(commitSha);
+        }
+    }
+
+    static final class BranchState {
+        private final String headSha;
+        private volatile long checkedAt;
+        private final Map<String, LatestVersion> latestVersions = new 
ConcurrentHashMap<>();
+        private final Map<String, RegisteredFlow> flows = new 
ConcurrentHashMap<>();
+
+        private BranchState(final String headSha, final long checkedAt) {
+            this.headSha = headSha;
+            this.checkedAt = checkedAt;
+        }
+
+        /**
+         * @return the latest version read for the file, empty when it was not 
read since the head of the branch changed
+         */
+        Optional<LatestVersion> getLatestVersion(final String path) {
+            return Optional.ofNullable(latestVersions.get(path));
+        }
+
+        void putLatestVersion(final String path, final Optional<String> 
commitSha) {
+            latestVersions.put(path, new 
LatestVersion(commitSha.orElse(null)));
+        }
+
+        Optional<RegisteredFlow> getFlow(final String path) {
+            return Optional.ofNullable(flows.get(path)).map(BranchState::copy);
+        }
+
+        void putFlow(final String path, final RegisteredFlow flow) {
+            flows.put(path, copy(flow));
+        }
+
+        private void remove(final String path) {
+            latestVersions.remove(path);
+            flows.remove(path);
+        }
+
+        private void retainUnchanged(final BranchState previousState, final 
Set<String> changedPaths) {
+            previousState.latestVersions.forEach((path, latestVersion) -> {
+                if (!changedPaths.contains(path)) {
+                    latestVersions.put(path, latestVersion);
+                }
+            });
+            previousState.flows.forEach((path, flow) -> {
+                if (!changedPaths.contains(path)) {
+                    flows.put(path, flow);
+                }
+            });
+        }
+
+        // Callers change the returned flow, so the cache keeps and hands out 
copies
+        private static RegisteredFlow copy(final RegisteredFlow flow) {
+            final RegisteredFlow copy = new RegisteredFlow();
+            copy.setIdentifier(flow.getIdentifier());
+            copy.setName(flow.getName());
+            copy.setDescription(flow.getDescription());
+            copy.setBranch(flow.getBranch());
+            copy.setBucketIdentifier(flow.getBucketIdentifier());
+            copy.setBucketName(flow.getBucketName());
+            copy.setCreatedTimestamp(flow.getCreatedTimestamp());
+            copy.setLastModifiedTimestamp(flow.getLastModifiedTimestamp());
+            copy.setPermissions(flow.getPermissions());

Review Comment:
   Not safe, thanks. 9c941684 copies the permissions and version info too, on 
put and on get; `testFlowPermissionsAndVersionInfoAreCopied` changes both the 
original and the returned objects and checks the cached flow keeps its values.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);

Review Comment:
   Good point, and skipping the cache for that one synchronization would need a 
way to tell it apart from the periodic one, which the registry client does not 
have. Instead, 15b75342 removes the 10-second window: the branch head is 
checked before every answer, so node B sees the commit from node A on that very 
synchronization. To keep that cheap, the GitHub client sends the head check as 
a conditional request (`ConditionalBranchReferenceConnector`, If-None-Match 
with the previous ETag); while the head is unchanged GitHub answers 304, which 
does not count against the primary rate limit. Checked against api.github.com 
with this client: one head check cost 7 requests in total, 6 of them building 
the client, and 51 checks cost 8.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -561,7 +572,15 @@ public Optional<String> getLatestVersion(final 
FlowRegistryClientConfigurationCo
         final String branch = flowLocation.getBranch();
         final String filePath = getSnapshotFilePath(flowLocation);
 
-        return repositoryClient.getLatestCommitSha(filePath, branch);
+        final Optional<GitBranchStateCache.BranchState> branchState = 
branchStateCache.getState(repositoryClient, branch);
+        final Optional<GitBranchStateCache.LatestVersion> cachedLatestVersion 
= branchState.flatMap(state -> state.getLatestVersion(filePath));
+        if (cachedLatestVersion.isPresent()) {
+            return cachedLatestVersion.get().toOptional();
+        }
+
+        final Optional<String> latestVersion = 
repositoryClient.getLatestCommitSha(filePath, branch);

Review Comment:
   Yes, done in 15b75342: when a state exists, the content is read with 
`getContentFromCommit(path, headSha)` and the latest version with the new 
`getLatestCommitShaAtCommit(path, headSha)` (GitHub: 
`queryCommits().from(headSha)`), so every cached value matches the head of the 
state that holds it. Without a state (clients that do not report a head) the 
reads are unchanged.



##########
nifi-extension-bundles/nifi-github-bundle/nifi-github-extensions/src/main/java/org/apache/nifi/github/GitHubRepositoryClient.java:
##########
@@ -379,13 +385,105 @@ static Optional<String> getLatestCommitSha(final 
GHRepository repository, final
             return commits.hasNext() ? Optional.of(commits.next().getSHA1()) : 
Optional.empty();
         } catch (final GHException e) {
             // Iterators wrap request failures, such as Not Found for an 
unknown branch, which callers handle as IOException
-            if (e.getCause() instanceof IOException ioException) {
+            if (e.getCause() instanceof final IOException ioException) {
                 throw ioException;
             }
             throw e;
         }
     }
 
+    /**
+     * Gets the SHA of the commit at the head of the given branch with a 
single request.
+     *
+     * @param branch the branch
+     * @return the SHA of the head commit
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling 
GitHub, or the branch does not exist
+     */
+    @Override
+    public Optional<String> getBranchHeadSha(final String branch) throws 
IOException, FlowRegistryException {
+        final String branchRef = BRANCH_REF_PATTERN.formatted(branch);
+        logger.debug("Getting head commit of branch [{}] in repo [{}]", 
branch, repository.getName());
+
+        return execute(() -> {
+            try {
+                return 
Optional.of(repository.getRef(branchRef).getObject().getSha());
+            } catch (final FileNotFoundException fnf) {
+                throw new FlowRegistryException("Branch [" + branchRef + "] 
not found", fnf);
+            }
+        });
+    }
+
+    /**
+     * Gets the paths touched by the commits from one commit to a later one. 
One compare request answers when a single
+     * commit separates them; otherwise each commit in the range is read, 
because the compared files are only the net
+     * difference and miss a file changed and then reverted within the range.
+     *
+     * @param fromCommitSha the SHA of the earlier commit
+     * @param toCommitSha the SHA of the later commit
+     * @return the touched paths relative to the repository path, or empty 
when the later commit does not descend from
+     *         the earlier one, or the range has more commits or files than 
are read
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling GitHub
+     */
+    @Override
+    public Optional<Set<String>> getChangedPaths(final String fromCommitSha, 
final String toCommitSha) throws IOException, FlowRegistryException {
+        logger.debug("Getting paths changed from [{}] to [{}] in repo [{}]", 
fromCommitSha, toCommitSha, repository.getName());
+        return execute(() -> 
getChangedPaths(repository.getCompare(fromCommitSha, toCommitSha), sha -> 
repository.getCommit(sha).listFiles(), repoPath));
+    }
+
+    @FunctionalInterface
+    interface CommitFilesReader {
+        Iterable<GHCommit.File> getFiles(String commitSha) throws IOException;
+    }
+
+    static Optional<Set<String>> getChangedPaths(final GHCompare compare, 
final CommitFilesReader commitFilesReader, final String repoPath) throws 
IOException {
+        final GHCompare.Status status = compare.getStatus();
+        if (status != GHCompare.Status.ahead && status != 
GHCompare.Status.identical) {
+            return Optional.empty();
+        }
+
+        final GHCompare.Commit[] commits = compare.getCommits();
+        if (commits.length != compare.getTotalCommits() || commits.length > 
COMPARE_COMMITS_LIMIT) {
+            return Optional.empty();
+        }
+
+        final Set<String> changedPaths = new HashSet<>();
+        if (commits.length <= 1) {

Review Comment:
   Agreed, simpler and safer. 15b75342 resolves the paths only when the compare 
status is ahead and `aheadBy` is 1, from the compare's file list (300 or more 
files still drops the branch); any other range drops the whole branch. The 
per-commit listing and its limit are gone. With the head checked on every call, 
two consecutive heads are usually one commit apart.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/client/GitRepositoryClient.java:
##########
@@ -94,6 +94,32 @@ default Optional<String> getLatestCommitSha(String path, 
String branch) throws I
         return getCommits(path, 
branch).stream().findFirst().map(GitCommit::id);
     }
 
+    /**
+     * Retrieves the SHA of the commit at the head of the given branch. 
Clients that support it let the registry client
+     * answer repeated questions about unchanged flows without further 
requests.
+     *
+     * @param branch the branch name
+     * @return the SHA of the head commit, or empty if the client does not 
support it
+     * @throws IOException if an I/O error occurs
+     * @throws FlowRegistryException if a non-I/O error occurs
+     */
+    default Optional<String> getBranchHeadSha(String branch) throws 
IOException, FlowRegistryException {
+        return Optional.empty();
+    }
+
+    /**
+     * Retrieves the paths of the files changed from one commit to a later 
one, relative to the repository path of the client.
+     *
+     * @param fromCommitSha the SHA of the earlier commit
+     * @param toCommitSha the SHA of the later commit
+     * @return the changed paths, or empty if they cannot be determined, for 
example when the history diverged
+     * @throws IOException if an I/O error occurs
+     * @throws FlowRegistryException if a non-I/O error occurs
+     */
+    default Optional<Set<String>> getChangedPaths(String fromCommitSha, String 
toCommitSha) throws IOException, FlowRegistryException {

Review Comment:
   Yes, updated in 15b75342: the Javadoc now says it returns the paths touched 
by any commit in the range, including both paths of a rename, that it is not 
the net difference (a file changed and reverted within the range is included), 
and empty when this cannot be determined. `getBranchHeadSha` also says that a 
client returning a head must implement `getChangedPaths` and 
`getLatestCommitShaAtCommit`, and that the check should be cheap while the head 
is unchanged.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
+
+    private final LongSupplier nanoTimeSupplier;
+    private final Map<String, BranchState> branchStates = new 
ConcurrentHashMap<>();
+
+    GitBranchStateCache() {
+        this(System::nanoTime);
+    }
+
+    GitBranchStateCache(final LongSupplier nanoTimeSupplier) {
+        this.nanoTimeSupplier = nanoTimeSupplier;
+    }
+
+    /**
+     * Gets the state of the branch, checking its head when the last check is 
older than the check interval.
+     * When the head moved, values cached for files changed between the two 
heads are dropped; when those files
+     * cannot be determined, every value cached for the branch is dropped.
+     *
+     * @param repositoryClient the repository client
+     * @param branch the branch
+     * @return the state of the branch, or empty when the repository client 
cannot report the branch head
+     */
+    Optional<BranchState> getState(final GitRepositoryClient repositoryClient, 
final String branch) throws IOException, FlowRegistryException {
+        final long now = nanoTimeSupplier.getAsLong();
+        final BranchState state = branchStates.get(branch);
+        if (state != null && now - state.checkedAt < 
HEAD_CHECK_INTERVAL_NANOS) {
+            return Optional.of(state);
+        }
+
+        final Optional<String> headSha = 
repositoryClient.getBranchHeadSha(branch);
+        if (headSha.isEmpty()) {
+            branchStates.remove(branch);
+            return Optional.empty();
+        }
+
+        if (state != null && state.headSha.equals(headSha.get())) {
+            state.checkedAt = now;
+            return Optional.of(state);
+        }
+
+        final BranchState currentState = new BranchState(headSha.get(), now);
+        if (state != null) {
+            getChangedPaths(repositoryClient, state.headSha, 
currentState.headSha).ifPresent(paths -> currentState.retainUnchanged(state, 
paths));
+        }
+        // Values put into the replaced state by requests still running are 
dropped with it
+        branchStates.put(branch, currentState);
+        return Optional.of(currentState);
+    }
+
+    // A failed comparison, such as Not Found when the previous head was 
removed by a force push, leaves the changed
+    // files unknown: the new state starts empty instead of the failure 
keeping the previous state for every later call
+    private static Optional<Set<String>> getChangedPaths(final 
GitRepositoryClient repositoryClient, final String fromCommitSha, final String 
toCommitSha) {
+        try {
+            return repositoryClient.getChangedPaths(fromCommitSha, 
toCommitSha);
+        } catch (final IOException | FlowRegistryException e) {
+            return Optional.empty();
+        }
+    }
+
+    /**
+     * Drops the values cached for the given file, after this client changed 
it.
+     *
+     * @param branch the branch
+     * @param path the path of the file
+     */
+    void invalidate(final String branch, final String path) {
+        final BranchState state = branchStates.get(branch);
+        if (state != null) {
+            state.remove(path);
+        }
+    }
+
+    void clear() {

Review Comment:
   Tied to the client instance in 15b75342: each branch state keeps the 
repository client it was read with and is only used with that same instance. A 
call still running with the old client can put a state back after `clear()`, 
but the new client never uses it and replaces it on its first call. 
`testStateIsNotUsedWithOtherRepositoryClient` covers both directions.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -503,6 +513,7 @@ public RegisteredFlowSnapshot registerFlowSnapshot(final 
FlowRegistryClientConfi
                 .build();
 
         final String createContentCommitSha = 
repositoryClient.createContent(createContentRequest);
+        branchStateCache.invalidate(branch, filePath);

Review Comment:
   The invalidate calls are gone in 15b75342: since the head is checked on 
every call, a write moves the head and the next read compares the two heads and 
drops the written path. `writeIsVisibleOnNextRead` registers a flow through the 
client with a mocked repository whose head moves on `createContent`, and checks 
the next `getLatestVersion` and `getFlow` return the new values. I checked the 
tests fail when the head comparison, the dropping of touched paths, the 
ordering of head checks or the client instance check is broken.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to