pvillard31 commented on code in PR #11926:
URL: https://github.com/apache/nifi/pull/11926#discussion_r4600235665
##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/client/GitRepositoryClient.java:
##########
@@ -94,6 +94,32 @@ default Optional<String> getLatestCommitSha(String path,
String branch) throws I
return getCommits(path,
branch).stream().findFirst().map(GitCommit::id);
}
+ /**
+ * Retrieves the SHA of the commit at the head of the given branch.
Clients that support it let the registry client
+ * answer repeated questions about unchanged flows without further
requests.
+ *
+ * @param branch the branch name
+ * @return the SHA of the head commit, or empty if the client does not
support it
+ * @throws IOException if an I/O error occurs
+ * @throws FlowRegistryException if a non-I/O error occurs
+ */
+ default Optional<String> getBranchHeadSha(String branch) throws
IOException, FlowRegistryException {
+ return Optional.empty();
+ }
+
+ /**
+ * Retrieves the paths of the files changed from one commit to a later
one, relative to the repository path of the client.
+ *
+ * @param fromCommitSha the SHA of the earlier commit
+ * @param toCommitSha the SHA of the later commit
+ * @return the changed paths, or empty if they cannot be determined, for
example when the history diverged
+ * @throws IOException if an I/O error occurs
+ * @throws FlowRegistryException if a non-I/O error occurs
+ */
+ default Optional<Set<String>> getChangedPaths(String fromCommitSha, String
toCommitSha) throws IOException, FlowRegistryException {
Review Comment:
Since other clients will implement this method, should the Javadoc say that
it returns the paths touched by any commit in the range (including the old path
of a rename), not only the net difference, and empty when this cannot be
determined?
##########
nifi-extension-bundles/nifi-github-bundle/nifi-github-extensions/src/main/java/org/apache/nifi/github/GitHubRepositoryClient.java:
##########
@@ -379,13 +385,105 @@ static Optional<String> getLatestCommitSha(final
GHRepository repository, final
return commits.hasNext() ? Optional.of(commits.next().getSHA1()) :
Optional.empty();
} catch (final GHException e) {
// Iterators wrap request failures, such as Not Found for an
unknown branch, which callers handle as IOException
- if (e.getCause() instanceof IOException ioException) {
+ if (e.getCause() instanceof final IOException ioException) {
throw ioException;
}
throw e;
}
}
+ /**
+ * Gets the SHA of the commit at the head of the given branch with a
single request.
+ *
+ * @param branch the branch
+ * @return the SHA of the head commit
+ *
+ * @throws IOException if an I/O error happens calling GitHub
+ * @throws FlowRegistryException if a non I/O error happens calling
GitHub, or the branch does not exist
+ */
+ @Override
+ public Optional<String> getBranchHeadSha(final String branch) throws
IOException, FlowRegistryException {
+ final String branchRef = BRANCH_REF_PATTERN.formatted(branch);
+ logger.debug("Getting head commit of branch [{}] in repo [{}]",
branch, repository.getName());
+
+ return execute(() -> {
+ try {
+ return
Optional.of(repository.getRef(branchRef).getObject().getSha());
+ } catch (final FileNotFoundException fnf) {
+ throw new FlowRegistryException("Branch [" + branchRef + "]
not found", fnf);
+ }
+ });
+ }
+
+ /**
+ * Gets the paths touched by the commits from one commit to a later one.
One compare request answers when a single
+ * commit separates them; otherwise each commit in the range is read,
because the compared files are only the net
+ * difference and miss a file changed and then reverted within the range.
+ *
+ * @param fromCommitSha the SHA of the earlier commit
+ * @param toCommitSha the SHA of the later commit
+ * @return the touched paths relative to the repository path, or empty
when the later commit does not descend from
+ * the earlier one, or the range has more commits or files than
are read
+ *
+ * @throws IOException if an I/O error happens calling GitHub
+ * @throws FlowRegistryException if a non I/O error happens calling GitHub
+ */
+ @Override
+ public Optional<Set<String>> getChangedPaths(final String fromCommitSha,
final String toCommitSha) throws IOException, FlowRegistryException {
+ logger.debug("Getting paths changed from [{}] to [{}] in repo [{}]",
fromCommitSha, toCommitSha, repository.getName());
+ return execute(() ->
getChangedPaths(repository.getCompare(fromCommitSha, toCommitSha), sha ->
repository.getCommit(sha).listFiles(), repoPath));
+ }
+
+ @FunctionalInterface
+ interface CommitFilesReader {
+ Iterable<GHCommit.File> getFiles(String commitSha) throws IOException;
+ }
+
+ static Optional<Set<String>> getChangedPaths(final GHCompare compare,
final CommitFilesReader commitFilesReader, final String repoPath) throws
IOException {
+ final GHCompare.Status status = compare.getStatus();
+ if (status != GHCompare.Status.ahead && status !=
GHCompare.Status.identical) {
+ return Optional.empty();
+ }
+
+ final GHCompare.Commit[] commits = compare.getCommits();
+ if (commits.length != compare.getTotalCommits() || commits.length >
COMPARE_COMMITS_LIMIT) {
+ return Optional.empty();
+ }
+
+ final Set<String> changedPaths = new HashSet<>();
+ if (commits.length <= 1) {
Review Comment:
Should getChangedPaths only answer when the compare is ahead by exactly one
commit, and drop the whole branch in every other case? This keeps one request
for the common case and removes the per commit file listing, which has no guard
when the file list of a commit is truncated.
##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval,
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+ static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
+
+ private final LongSupplier nanoTimeSupplier;
+ private final Map<String, BranchState> branchStates = new
ConcurrentHashMap<>();
+
+ GitBranchStateCache() {
+ this(System::nanoTime);
+ }
+
+ GitBranchStateCache(final LongSupplier nanoTimeSupplier) {
+ this.nanoTimeSupplier = nanoTimeSupplier;
+ }
+
+ /**
+ * Gets the state of the branch, checking its head when the last check is
older than the check interval.
+ * When the head moved, values cached for files changed between the two
heads are dropped; when those files
+ * cannot be determined, every value cached for the branch is dropped.
+ *
+ * @param repositoryClient the repository client
+ * @param branch the branch
+ * @return the state of the branch, or empty when the repository client
cannot report the branch head
+ */
+ Optional<BranchState> getState(final GitRepositoryClient repositoryClient,
final String branch) throws IOException, FlowRegistryException {
+ final long now = nanoTimeSupplier.getAsLong();
+ final BranchState state = branchStates.get(branch);
+ if (state != null && now - state.checkedAt <
HEAD_CHECK_INTERVAL_NANOS) {
+ return Optional.of(state);
+ }
+
+ final Optional<String> headSha =
repositoryClient.getBranchHeadSha(branch);
+ if (headSha.isEmpty()) {
+ branchStates.remove(branch);
+ return Optional.empty();
+ }
+
+ if (state != null && state.headSha.equals(headSha.get())) {
+ state.checkedAt = now;
+ return Optional.of(state);
+ }
+
+ final BranchState currentState = new BranchState(headSha.get(), now);
+ if (state != null) {
+ getChangedPaths(repositoryClient, state.headSha,
currentState.headSha).ifPresent(paths -> currentState.retainUnchanged(state,
paths));
+ }
+ // Values put into the replaced state by requests still running are
dropped with it
+ branchStates.put(branch, currentState);
+ return Optional.of(currentState);
+ }
+
+ // A failed comparison, such as Not Found when the previous head was
removed by a force push, leaves the changed
+ // files unknown: the new state starts empty instead of the failure
keeping the previous state for every later call
+ private static Optional<Set<String>> getChangedPaths(final
GitRepositoryClient repositoryClient, final String fromCommitSha, final String
toCommitSha) {
+ try {
+ return repositoryClient.getChangedPaths(fromCommitSha,
toCommitSha);
+ } catch (final IOException | FlowRegistryException e) {
+ return Optional.empty();
+ }
+ }
+
+ /**
+ * Drops the values cached for the given file, after this client changed
it.
+ *
+ * @param branch the branch
+ * @param path the path of the file
+ */
+ void invalidate(final String branch, final String path) {
+ final BranchState state = branchStates.get(branch);
+ if (state != null) {
+ state.remove(path);
+ }
+ }
+
+ void clear() {
Review Comment:
clear() runs under the client lock, but a call already in progress with the
old repository client can put entries back right after it. Should the cache be
tied to the repository client instance, or use a generation counter, so a
change of Repository Path cannot leave entries from the old configuration?
##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -561,7 +572,15 @@ public Optional<String> getLatestVersion(final
FlowRegistryClientConfigurationCo
final String branch = flowLocation.getBranch();
final String filePath = getSnapshotFilePath(flowLocation);
- return repositoryClient.getLatestCommitSha(filePath, branch);
+ final Optional<GitBranchStateCache.BranchState> branchState =
branchStateCache.getState(repositoryClient, branch);
+ final Optional<GitBranchStateCache.LatestVersion> cachedLatestVersion
= branchState.flatMap(state -> state.getLatestVersion(filePath));
+ if (cachedLatestVersion.isPresent()) {
+ return cachedLatestVersion.get().toOptional();
+ }
+
+ final Optional<String> latestVersion =
repositoryClient.getLatestCommitSha(filePath, branch);
Review Comment:
Values are read at the current branch tip but stored under the head SHA of
the state, so a force push or reset inside the 10 seconds can keep values from
a commit that is no longer on the branch. Should the reads (latest commit and
content) use the head SHA of the state, for example
queryCommits().from(headSha), so each cached value matches its state?
##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.flow.git;
+
+import org.apache.nifi.registry.flow.FlowRegistryException;
+import org.apache.nifi.registry.flow.FlowRegistryPermissions;
+import org.apache.nifi.registry.flow.RegisteredFlow;
+import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo;
+import org.apache.nifi.registry.flow.git.client.GitRepositoryClient;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.function.LongSupplier;
+
+/**
+ * Remembers, per branch, what was read for each flow file while the head of
the branch stays the same.
+ * <p>
+ * The framework asks for the flow and its latest version for every versioned
Process Group on every
+ * registry synchronization. Without this cache each of those questions costs
requests to the Git provider,
+ * so the cost of a synchronization grows with the number of versioned Process
Groups. With it, a
+ * synchronization costs one request for the branch head per check interval,
plus requests for the files
+ * that changed since the previous head.
+ */
+final class GitBranchStateCache {
+
+ static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10);
Review Comment:
In a cluster, a commit on node A sends the new version to node B, which runs
synchronizeWithFlowRegistry right away; if B checked the head in the last 10
seconds, it reads the old latest version from its cache and marks the group
stale until the next sync (30 min by default), which also blocks a commit from
B. Should this synchronization skip the cache, or is there another way to avoid
it?
##########
nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java:
##########
@@ -503,6 +513,7 @@ public RegisteredFlowSnapshot registerFlowSnapshot(final
FlowRegistryClientConfi
.build();
final String createContentCommitSha =
repositoryClient.createContent(createContentRequest);
+ branchStateCache.invalidate(branch, filePath);
Review Comment:
Removing any of the three invalidate calls does not make a test fail,
because the mocked clients return an empty head SHA. Could an existing test
return a head SHA so it checks that a write is visible on the next read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]