pvillard31 commented on code in PR #11926:
URL: https://github.com/apache/nifi/pull/11926#discussion_r4600235665


##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/client/GitRepositoryClient.java:
##########
@@ -94,6 +94,32 @@ default Optional<String> getLatestCommitSha(String path, 
String branch) throws I
         return getCommits(path, 
branch).stream().findFirst().map(GitCommit::id);
     }
 
+    /**
+     * Retrieves the SHA of the commit at the head of the given branch. 
Clients that support it let the registry client
+     * answer repeated questions about unchanged flows without further 
requests.
+     *
+     * @param branch the branch name
+     * @return the SHA of the head commit, or empty if the client does not 
support it
+     * @throws IOException if an I/O error occurs
+     * @throws FlowRegistryException if a non-I/O error occurs
+     */
+    default Optional<String> getBranchHeadSha(String branch) throws 
IOException, FlowRegistryException {
+        return Optional.empty();
+    }
+
+    /**
+     * Retrieves the paths of the files changed from one commit to a later 
one, relative to the repository path of the client.
+     *
+     * @param fromCommitSha the SHA of the earlier commit
+     * @param toCommitSha the SHA of the later commit
+     * @return the changed paths, or empty if they cannot be determined, for 
example when the history diverged
+     * @throws IOException if an I/O error occurs
+     * @throws FlowRegistryException if a non-I/O error occurs
+     */
+    default Optional<Set<String>> getChangedPaths(String fromCommitSha, String 
toCommitSha) throws IOException, FlowRegistryException {

Review Comment:
   Since other clients will implement this method, should the Javadoc say that 
it returns the paths touched by any commit in the range (including the old path 
of a rename), not only the net difference, and empty when this cannot be 
determined?



##########
nifi-extension-bundles/nifi-github-bundle/nifi-github-extensions/src/main/java/org/apache/nifi/github/GitHubRepositoryClient.java:
##########
@@ -379,13 +385,105 @@ static Optional<String> getLatestCommitSha(final 
GHRepository repository, final
             return commits.hasNext() ? Optional.of(commits.next().getSHA1()) : 
Optional.empty();
         } catch (final GHException e) {
             // Iterators wrap request failures, such as Not Found for an 
unknown branch, which callers handle as IOException
-            if (e.getCause() instanceof IOException ioException) {
+            if (e.getCause() instanceof final IOException ioException) {
                 throw ioException;
             }
             throw e;
         }
     }
 
+    /**
+     * Gets the SHA of the commit at the head of the given branch with a 
single request.
+     *
+     * @param branch the branch
+     * @return the SHA of the head commit
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling 
GitHub, or the branch does not exist
+     */
+    @Override
+    public Optional<String> getBranchHeadSha(final String branch) throws 
IOException, FlowRegistryException {
+        final String branchRef = BRANCH_REF_PATTERN.formatted(branch);
+        logger.debug("Getting head commit of branch [{}] in repo [{}]", 
branch, repository.getName());
+
+        return execute(() -> {
+            try {
+                return 
Optional.of(repository.getRef(branchRef).getObject().getSha());
+            } catch (final FileNotFoundException fnf) {
+                throw new FlowRegistryException("Branch [" + branchRef + "] 
not found", fnf);
+            }
+        });
+    }
+
+    /**
+     * Gets the paths touched by the commits from one commit to a later one. 
One compare request answers when a single
+     * commit separates them; otherwise each commit in the range is read, 
because the compared files are only the net
+     * difference and miss a file changed and then reverted within the range.
+     *
+     * @param fromCommitSha the SHA of the earlier commit
+     * @param toCommitSha the SHA of the later commit
+     * @return the touched paths relative to the repository path, or empty 
when the later commit does not descend from
+     *         the earlier one, or the range has more commits or files than 
are read
+     *
+     * @throws IOException if an I/O error happens calling GitHub
+     * @throws FlowRegistryException if a non I/O error happens calling GitHub
+     */
+    @Override
+    public Optional<Set<String>> getChangedPaths(final String fromCommitSha, 
final String toCommitSha) throws IOException, FlowRegistryException {
+        logger.debug("Getting paths changed from [{}] to [{}] in repo [{}]", 
fromCommitSha, toCommitSha, repository.getName());
+        return execute(() -> 
getChangedPaths(repository.getCompare(fromCommitSha, toCommitSha), sha -> 
repository.getCommit(sha).listFiles(), repoPath));
+    }
+
+    @FunctionalInterface
+    interface CommitFilesReader {
+        Iterable<GHCommit.File> getFiles(String commitSha) throws IOException;
+    }
+
+    static Optional<Set<String>> getChangedPaths(final GHCompare compare, 
final CommitFilesReader commitFilesReader, final String repoPath) throws 
IOException {
+        final GHCompare.Status status = compare.getStatus();
+        if (status != GHCompare.Status.ahead && status != 
GHCompare.Status.identical) {
+            return Optional.empty();
+        }
+
+        final GHCompare.Commit[] commits = compare.getCommits();
+        if (commits.length != compare.getTotalCommits() || commits.length > 
COMPARE_COMMITS_LIMIT) {
+            return Optional.empty();
+        }
+
+        final Set<String> changedPaths = new HashSet<>();
+        if (commits.length <= 1) {

Review Comment:
   Should getChangedPaths only answer when the compare is ahead by exactly one 
commit, and drop the whole branch in every other case? This keeps one request 
for the common case and removes the per commit file listing, which has no guard 
when the file list of a commit is truncated.



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
+
+    private final LongSupplier nanoTimeSupplier;
+    private final Map<String, BranchState> branchStates = new 
ConcurrentHashMap<>();
+
+    GitBranchStateCache() {
+        this(System::nanoTime);
+    }
+
+    GitBranchStateCache(final LongSupplier nanoTimeSupplier) {
+        this.nanoTimeSupplier = nanoTimeSupplier;
+    }
+
+    /**
+     * Gets the state of the branch, checking its head when the last check is 
older than the check interval.
+     * When the head moved, values cached for files changed between the two 
heads are dropped; when those files
+     * cannot be determined, every value cached for the branch is dropped.
+     *
+     * @param repositoryClient the repository client
+     * @param branch the branch
+     * @return the state of the branch, or empty when the repository client 
cannot report the branch head
+     */
+    Optional<BranchState> getState(final GitRepositoryClient repositoryClient, 
final String branch) throws IOException, FlowRegistryException {
+        final long now = nanoTimeSupplier.getAsLong();
+        final BranchState state = branchStates.get(branch);
+        if (state != null && now - state.checkedAt < 
HEAD_CHECK_INTERVAL_NANOS) {
+            return Optional.of(state);
+        }
+
+        final Optional<String> headSha = 
repositoryClient.getBranchHeadSha(branch);
+        if (headSha.isEmpty()) {
+            branchStates.remove(branch);
+            return Optional.empty();
+        }
+
+        if (state != null && state.headSha.equals(headSha.get())) {
+            state.checkedAt = now;
+            return Optional.of(state);
+        }
+
+        final BranchState currentState = new BranchState(headSha.get(), now);
+        if (state != null) {
+            getChangedPaths(repositoryClient, state.headSha, 
currentState.headSha).ifPresent(paths -> currentState.retainUnchanged(state, 
paths));
+        }
+        // Values put into the replaced state by requests still running are 
dropped with it
+        branchStates.put(branch, currentState);
+        return Optional.of(currentState);
+    }
+
+    // A failed comparison, such as Not Found when the previous head was 
removed by a force push, leaves the changed
+    // files unknown: the new state starts empty instead of the failure 
keeping the previous state for every later call
+    private static Optional<Set<String>> getChangedPaths(final 
GitRepositoryClient repositoryClient, final String fromCommitSha, final String 
toCommitSha) {
+        try {
+            return repositoryClient.getChangedPaths(fromCommitSha, 
toCommitSha);
+        } catch (final IOException | FlowRegistryException e) {
+            return Optional.empty();
+        }
+    }
+
+    /**
+     * Drops the values cached for the given file, after this client changed 
it.
+     *
+     * @param branch the branch
+     * @param path the path of the file
+     */
+    void invalidate(final String branch, final String path) {
+        final BranchState state = branchStates.get(branch);
+        if (state != null) {
+            state.remove(path);
+        }
+    }
+
+    void clear() {

Review Comment:
   clear() runs under the client lock, but a call already in progress with the 
old repository client can put entries back right after it. Should the cache be 
tied to the repository client instance, or use a generation counter, so a 
change of Repository Path cannot leave entries from the old configuration?



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -561,7 +572,15 @@ public Optional<String> getLatestVersion(final 
FlowRegistryClientConfigurationCo
         final String branch = flowLocation.getBranch();
         final String filePath = getSnapshotFilePath(flowLocation);
 
-        return repositoryClient.getLatestCommitSha(filePath, branch);
+        final Optional<GitBranchStateCache.BranchState> branchState = 
branchStateCache.getState(repositoryClient, branch);
+        final Optional<GitBranchStateCache.LatestVersion> cachedLatestVersion 
= branchState.flatMap(state -> state.getLatestVersion(filePath));
+        if (cachedLatestVersion.isPresent()) {
+            return cachedLatestVersion.get().toOptional();
+        }
+
+        final Optional<String> latestVersion = 
repositoryClient.getLatestCommitSha(filePath, branch);

Review Comment:
   Values are read at the current branch tip but stored under the head SHA of 
the state, so a force push or reset inside the 10 seconds can keep values from 
a commit that is no longer on the branch. Should the reads (latest commit and 
content) use the head SHA of the state, for example 
queryCommits().from(headSha), so each cached value matches its state?



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of 
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned 
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs 
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process 
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval, 
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+    static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);

Review Comment:
   In a cluster, a commit on node A sends the new version to node B, which runs 
synchronizeWithFlowRegistry right away; if B checked the head in the last 10 
seconds, it reads the old latest version from its cache and marks the group 
stale until the next sync (30 min by default), which also blocks a commit from 
B. Should this synchronization skip the cache, or is there another way to avoid 
it?



##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -503,6 +513,7 @@ public RegisteredFlowSnapshot registerFlowSnapshot(final 
FlowRegistryClientConfi
                 .build();
 
         final String createContentCommitSha = 
repositoryClient.createContent(createContentRequest);
+        branchStateCache.invalidate(branch, filePath);

Review Comment:
   Removing any of the three invalidate calls does not make a test fail, 
because the mocked clients return an empty head SHA. Could an existing test 
return a head SHA so it checks that a write is visible on the next read?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to