Rarasinhoo commented on code in PR #11926: URL: https://github.com/apache/nifi/pull/11926#discussion_r4600931849
########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java: ########## @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.registry.flow.git; + +import org.apache.nifi.registry.flow.FlowRegistryException; +import org.apache.nifi.registry.flow.FlowRegistryPermissions; +import org.apache.nifi.registry.flow.RegisteredFlow; +import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo; +import org.apache.nifi.registry.flow.git.client.GitRepositoryClient; + +import java.io.IOException; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; +import java.util.function.LongSupplier; + +/** + * Remembers, per branch, what was read for each flow file while the head of the branch stays the same. + * <p> + * The framework asks for the flow and its latest version for every versioned Process Group on every + * registry synchronization. Without this cache each of those questions costs requests to the Git provider, + * so the cost of a synchronization grows with the number of versioned Process Groups. With it, a + * synchronization costs one request for the branch head per check interval, plus requests for the files + * that changed since the previous head. + */ +final class GitBranchStateCache { + + static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10); Review Comment: Good point, and skipping the cache for that one synchronization would need a way to tell it apart from the periodic one, which the registry client does not have. Instead, ab965f36 removes the 10-second window: the branch head is checked before every answer, so node B sees the commit from node A on that very synchronization. To keep that cheap, the GitHub client sends the head check as a conditional request (`ConditionalBranchReferenceConnector`, If-None-Match with the previous ETag); while the head is unchanged GitHub answers 304, which does not count against the primary rate limit. Checked against api.github.com with this client: one head check cost 7 requests in total, 6 of them building the client, and 51 checks cost 8. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java: ########## @@ -561,7 +572,15 @@ public Optional<String> getLatestVersion(final FlowRegistryClientConfigurationCo final String branch = flowLocation.getBranch(); final String filePath = getSnapshotFilePath(flowLocation); - return repositoryClient.getLatestCommitSha(filePath, branch); + final Optional<GitBranchStateCache.BranchState> branchState = branchStateCache.getState(repositoryClient, branch); + final Optional<GitBranchStateCache.LatestVersion> cachedLatestVersion = branchState.flatMap(state -> state.getLatestVersion(filePath)); + if (cachedLatestVersion.isPresent()) { + return cachedLatestVersion.get().toOptional(); + } + + final Optional<String> latestVersion = repositoryClient.getLatestCommitSha(filePath, branch); Review Comment: Yes, done in ab965f36: when a state exists, the content is read with `getContentFromCommit(path, headSha)` and the latest version with the new `getLatestCommitShaAtCommit(path, headSha)` (GitHub: `queryCommits().from(headSha)`), so every cached value matches the head of the state that holds it. Without a state (clients that do not report a head) the reads are unchanged. ########## nifi-extension-bundles/nifi-github-bundle/nifi-github-extensions/src/main/java/org/apache/nifi/github/GitHubRepositoryClient.java: ########## @@ -379,13 +385,105 @@ static Optional<String> getLatestCommitSha(final GHRepository repository, final return commits.hasNext() ? Optional.of(commits.next().getSHA1()) : Optional.empty(); } catch (final GHException e) { // Iterators wrap request failures, such as Not Found for an unknown branch, which callers handle as IOException - if (e.getCause() instanceof IOException ioException) { + if (e.getCause() instanceof final IOException ioException) { throw ioException; } throw e; } } + /** + * Gets the SHA of the commit at the head of the given branch with a single request. + * + * @param branch the branch + * @return the SHA of the head commit + * + * @throws IOException if an I/O error happens calling GitHub + * @throws FlowRegistryException if a non I/O error happens calling GitHub, or the branch does not exist + */ + @Override + public Optional<String> getBranchHeadSha(final String branch) throws IOException, FlowRegistryException { + final String branchRef = BRANCH_REF_PATTERN.formatted(branch); + logger.debug("Getting head commit of branch [{}] in repo [{}]", branch, repository.getName()); + + return execute(() -> { + try { + return Optional.of(repository.getRef(branchRef).getObject().getSha()); + } catch (final FileNotFoundException fnf) { + throw new FlowRegistryException("Branch [" + branchRef + "] not found", fnf); + } + }); + } + + /** + * Gets the paths touched by the commits from one commit to a later one. One compare request answers when a single + * commit separates them; otherwise each commit in the range is read, because the compared files are only the net + * difference and miss a file changed and then reverted within the range. + * + * @param fromCommitSha the SHA of the earlier commit + * @param toCommitSha the SHA of the later commit + * @return the touched paths relative to the repository path, or empty when the later commit does not descend from + * the earlier one, or the range has more commits or files than are read + * + * @throws IOException if an I/O error happens calling GitHub + * @throws FlowRegistryException if a non I/O error happens calling GitHub + */ + @Override + public Optional<Set<String>> getChangedPaths(final String fromCommitSha, final String toCommitSha) throws IOException, FlowRegistryException { + logger.debug("Getting paths changed from [{}] to [{}] in repo [{}]", fromCommitSha, toCommitSha, repository.getName()); + return execute(() -> getChangedPaths(repository.getCompare(fromCommitSha, toCommitSha), sha -> repository.getCommit(sha).listFiles(), repoPath)); + } + + @FunctionalInterface + interface CommitFilesReader { + Iterable<GHCommit.File> getFiles(String commitSha) throws IOException; + } + + static Optional<Set<String>> getChangedPaths(final GHCompare compare, final CommitFilesReader commitFilesReader, final String repoPath) throws IOException { + final GHCompare.Status status = compare.getStatus(); + if (status != GHCompare.Status.ahead && status != GHCompare.Status.identical) { + return Optional.empty(); + } + + final GHCompare.Commit[] commits = compare.getCommits(); + if (commits.length != compare.getTotalCommits() || commits.length > COMPARE_COMMITS_LIMIT) { + return Optional.empty(); + } + + final Set<String> changedPaths = new HashSet<>(); + if (commits.length <= 1) { Review Comment: Agreed, simpler and safer. ab965f36 resolves the paths only when the compare status is ahead and `aheadBy` is 1, from the compare's file list (300 or more files still drops the branch); any other range drops the whole branch. The per-commit listing and its limit are gone. With the head checked on every call, two consecutive heads are usually one commit apart. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/client/GitRepositoryClient.java: ########## @@ -94,6 +94,32 @@ default Optional<String> getLatestCommitSha(String path, String branch) throws I return getCommits(path, branch).stream().findFirst().map(GitCommit::id); } + /** + * Retrieves the SHA of the commit at the head of the given branch. Clients that support it let the registry client + * answer repeated questions about unchanged flows without further requests. + * + * @param branch the branch name + * @return the SHA of the head commit, or empty if the client does not support it + * @throws IOException if an I/O error occurs + * @throws FlowRegistryException if a non-I/O error occurs + */ + default Optional<String> getBranchHeadSha(String branch) throws IOException, FlowRegistryException { + return Optional.empty(); + } + + /** + * Retrieves the paths of the files changed from one commit to a later one, relative to the repository path of the client. + * + * @param fromCommitSha the SHA of the earlier commit + * @param toCommitSha the SHA of the later commit + * @return the changed paths, or empty if they cannot be determined, for example when the history diverged + * @throws IOException if an I/O error occurs + * @throws FlowRegistryException if a non-I/O error occurs + */ + default Optional<Set<String>> getChangedPaths(String fromCommitSha, String toCommitSha) throws IOException, FlowRegistryException { Review Comment: Yes, updated in ab965f36: the Javadoc now says it returns the paths touched by any commit in the range, including both paths of a rename, that it is not the net difference (a file changed and reverted within the range is included), and empty when this cannot be determined. `getBranchHeadSha` also says that a client returning a head must implement `getChangedPaths` and `getLatestCommitShaAtCommit`, and that the check should be cheap while the head is unchanged. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java: ########## @@ -0,0 +1,214 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.registry.flow.git; + +import org.apache.nifi.registry.flow.FlowRegistryException; +import org.apache.nifi.registry.flow.FlowRegistryPermissions; +import org.apache.nifi.registry.flow.RegisteredFlow; +import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo; +import org.apache.nifi.registry.flow.git.client.GitRepositoryClient; + +import java.io.IOException; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; +import java.util.function.LongSupplier; + +/** + * Remembers, per branch, what was read for each flow file while the head of the branch stays the same. + * <p> + * The framework asks for the flow and its latest version for every versioned Process Group on every + * registry synchronization. Without this cache each of those questions costs requests to the Git provider, + * so the cost of a synchronization grows with the number of versioned Process Groups. With it, a + * synchronization costs one request for the branch head per check interval, plus requests for the files + * that changed since the previous head. + */ +final class GitBranchStateCache { + + static final long HEAD_CHECK_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(10); + + private final LongSupplier nanoTimeSupplier; + private final Map<String, BranchState> branchStates = new ConcurrentHashMap<>(); + + GitBranchStateCache() { + this(System::nanoTime); + } + + GitBranchStateCache(final LongSupplier nanoTimeSupplier) { + this.nanoTimeSupplier = nanoTimeSupplier; + } + + /** + * Gets the state of the branch, checking its head when the last check is older than the check interval. + * When the head moved, values cached for files changed between the two heads are dropped; when those files + * cannot be determined, every value cached for the branch is dropped. + * + * @param repositoryClient the repository client + * @param branch the branch + * @return the state of the branch, or empty when the repository client cannot report the branch head + */ + Optional<BranchState> getState(final GitRepositoryClient repositoryClient, final String branch) throws IOException, FlowRegistryException { + final long now = nanoTimeSupplier.getAsLong(); + final BranchState state = branchStates.get(branch); + if (state != null && now - state.checkedAt < HEAD_CHECK_INTERVAL_NANOS) { + return Optional.of(state); + } + + final Optional<String> headSha = repositoryClient.getBranchHeadSha(branch); + if (headSha.isEmpty()) { + branchStates.remove(branch); + return Optional.empty(); + } + + if (state != null && state.headSha.equals(headSha.get())) { + state.checkedAt = now; + return Optional.of(state); + } + + final BranchState currentState = new BranchState(headSha.get(), now); + if (state != null) { + getChangedPaths(repositoryClient, state.headSha, currentState.headSha).ifPresent(paths -> currentState.retainUnchanged(state, paths)); + } + // Values put into the replaced state by requests still running are dropped with it + branchStates.put(branch, currentState); + return Optional.of(currentState); + } + + // A failed comparison, such as Not Found when the previous head was removed by a force push, leaves the changed + // files unknown: the new state starts empty instead of the failure keeping the previous state for every later call + private static Optional<Set<String>> getChangedPaths(final GitRepositoryClient repositoryClient, final String fromCommitSha, final String toCommitSha) { + try { + return repositoryClient.getChangedPaths(fromCommitSha, toCommitSha); + } catch (final IOException | FlowRegistryException e) { + return Optional.empty(); + } + } + + /** + * Drops the values cached for the given file, after this client changed it. + * + * @param branch the branch + * @param path the path of the file + */ + void invalidate(final String branch, final String path) { + final BranchState state = branchStates.get(branch); + if (state != null) { + state.remove(path); + } + } + + void clear() { Review Comment: Tied to the client instance in ab965f36: each branch state keeps the repository client it was read with and is only used with that same instance. A call still running with the old client can put a state back after `clear()`, but the new client never uses it and replaces it on its first call. `testStateIsNotUsedWithOtherRepositoryClient` covers both directions. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClient.java: ########## @@ -503,6 +513,7 @@ public RegisteredFlowSnapshot registerFlowSnapshot(final FlowRegistryClientConfi .build(); final String createContentCommitSha = repositoryClient.createContent(createContentRequest); + branchStateCache.invalidate(branch, filePath); Review Comment: The invalidate calls are gone in ab965f36: since the head is checked on every call, a write moves the head and the next read compares the two heads and drops the written path. `writeIsVisibleOnNextRead` registers a flow through the client with a mocked repository whose head moves on `createContent`, and checks the next `getLatestVersion` and `getFlow` return the new values. I checked the tests fail when the head comparison, the dropping of touched paths, the ordering of head checks or the client instance check is broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
