Rarasinhoo commented on code in PR #11926: URL: https://github.com/apache/nifi/pull/11926#discussion_r4607354382
########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java: ########## @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.registry.flow.git; + +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.registry.flow.FlowRegistryException; +import org.apache.nifi.registry.flow.FlowRegistryPermissions; +import org.apache.nifi.registry.flow.RegisteredFlow; +import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo; +import org.apache.nifi.registry.flow.git.client.GitRepositoryClient; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Supplier; + +/** + * Remembers, per branch, what was read for each flow file at the head of the branch. + * <p> + * The framework asks for the flow and its latest version for every versioned Process Group on every + * registry synchronization. Without this cache each of those questions costs requests to the Git provider, + * so the cost of a synchronization grows with the number of versioned Process Groups. With it, each question + * costs a check of the branch head, which the repository client makes cheap while the head is unchanged, plus + * requests for the files touched since the previous head. + * <p> + * Each value is read at the head commit of the state that holds it, and a state is used only while the branch head + * is still that commit, so a value is never answered for another head. A state is also used only with the repository + * client instance it was read with, so values read before a change of configuration are never answered after it. + */ +final class GitBranchStateCache { + + private final Supplier<ComponentLog> loggerSupplier; + private final Map<String, BranchState> branchStates = new ConcurrentHashMap<>(); + private final Map<String, Lock> branchLocks = new ConcurrentHashMap<>(); + + GitBranchStateCache(final Supplier<ComponentLog> loggerSupplier) { + this.loggerSupplier = loggerSupplier; + } + + /** + * Gets the state of the branch at its current head. When the head moved since the state was read, the values for + * files touched between the two heads are dropped; when those files cannot be determined, every value is dropped. + * + * @param repositoryClient the repository client + * @param branch the branch + * @return the state of the branch, or empty when the repository client cannot report the branch head + */ + Optional<BranchState> getState(final GitRepositoryClient repositoryClient, final String branch) throws IOException, FlowRegistryException { + final Optional<String> headSha = repositoryClient.getBranchHeadSha(branch); + if (headSha.isEmpty()) { + return Optional.empty(); + } + + final BranchState state = branchStates.get(branch); + if (state != null && state.isReadWith(repositoryClient) && state.headSha.equals(headSha.get())) { + return Optional.of(state); + } + return Optional.of(moveState(repositoryClient, branch, headSha.get())); + } + + // Each call answers for the head it saw. Concurrent calls that saw different heads may move the state back and forth, + // which costs reads but never answers a value for another head. The lock only spares concurrent calls that saw the + // same new head from comparing the heads more than once. + private BranchState moveState(final GitRepositoryClient repositoryClient, final String branch, final String headSha) { + final Lock branchLock = branchLocks.computeIfAbsent(branch, key -> new ReentrantLock()); + branchLock.lock(); Review Comment: Moved outside, in e16486a7: the heads are compared before the state of the branch is replaced, and the new state is installed with `ConcurrentHashMap.compute`, so no call waits for another call's comparison. The values kept from the previous state stay valid whatever happened meanwhile, since the comparison is from that state's head. Concurrent calls that saw the same new head may each compare the heads; that only happens when the head moves. `testComparisonDoesNotHoldOtherCallsForTheBranch` holds a comparison and answers another call for the branch meanwhile; it times out on the previous code. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/main/java/org/apache/nifi/registry/flow/git/GitBranchStateCache.java: ########## @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.registry.flow.git; + +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.registry.flow.FlowRegistryException; +import org.apache.nifi.registry.flow.FlowRegistryPermissions; +import org.apache.nifi.registry.flow.RegisteredFlow; +import org.apache.nifi.registry.flow.RegisteredFlowVersionInfo; +import org.apache.nifi.registry.flow.git.client.GitRepositoryClient; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; +import java.util.function.Supplier; + +/** + * Remembers, per branch, what was read for each flow file at the head of the branch. + * <p> + * The framework asks for the flow and its latest version for every versioned Process Group on every + * registry synchronization. Without this cache each of those questions costs requests to the Git provider, + * so the cost of a synchronization grows with the number of versioned Process Groups. With it, each question + * costs a check of the branch head, which the repository client makes cheap while the head is unchanged, plus + * requests for the files touched since the previous head. + * <p> + * Each value is read at the head commit of the state that holds it, and a state is used only while the branch head + * is still that commit, so a value is never answered for another head. A state is also used only with the repository + * client instance it was read with, so values read before a change of configuration are never answered after it. + */ +final class GitBranchStateCache { + + private final Supplier<ComponentLog> loggerSupplier; + private final Map<String, BranchState> branchStates = new ConcurrentHashMap<>(); + private final Map<String, Lock> branchLocks = new ConcurrentHashMap<>(); + + GitBranchStateCache(final Supplier<ComponentLog> loggerSupplier) { + this.loggerSupplier = loggerSupplier; + } + + /** + * Gets the state of the branch at its current head. When the head moved since the state was read, the values for + * files touched between the two heads are dropped; when those files cannot be determined, every value is dropped. + * + * @param repositoryClient the repository client + * @param branch the branch + * @return the state of the branch, or empty when the repository client cannot report the branch head + */ + Optional<BranchState> getState(final GitRepositoryClient repositoryClient, final String branch) throws IOException, FlowRegistryException { + final Optional<String> headSha = repositoryClient.getBranchHeadSha(branch); + if (headSha.isEmpty()) { + return Optional.empty(); + } + + final BranchState state = branchStates.get(branch); + if (state != null && state.isReadWith(repositoryClient) && state.headSha.equals(headSha.get())) { + return Optional.of(state); + } + return Optional.of(moveState(repositoryClient, branch, headSha.get())); + } + + // Each call answers for the head it saw. Concurrent calls that saw different heads may move the state back and forth, + // which costs reads but never answers a value for another head. The lock only spares concurrent calls that saw the + // same new head from comparing the heads more than once. + private BranchState moveState(final GitRepositoryClient repositoryClient, final String branch, final String headSha) { + final Lock branchLock = branchLocks.computeIfAbsent(branch, key -> new ReentrantLock()); + branchLock.lock(); + try { + final BranchState state = branchStates.get(branch); + final boolean previousStateUsable = state != null && state.isReadWith(repositoryClient); + if (previousStateUsable && state.headSha.equals(headSha)) { + return state; + } + + final BranchState currentState = new BranchState(repositoryClient, headSha); + if (previousStateUsable) { + getChangedPaths(repositoryClient, branch, state.headSha, headSha).ifPresent(paths -> currentState.retainUnchanged(state, paths)); + } + branchStates.put(branch, currentState); + return currentState; + } finally { + branchLock.unlock(); + } + } + + // A failed comparison, such as Not Found when the previous head was removed by a force push, leaves the touched + // files unknown, so the new state starts empty + private Optional<Set<String>> getChangedPaths(final GitRepositoryClient repositoryClient, final String branch, final String fromCommitSha, final String toCommitSha) { + try { + return repositoryClient.getChangedPaths(fromCommitSha, toCommitSha); + } catch (final FileNotFoundException e) { + loggerSupplier.get().debug("Paths changed on branch [{}] from [{}] to [{}] not found, dropping every cached value", branch, fromCommitSha, toCommitSha, e); + return Optional.empty(); + } catch (final IOException | FlowRegistryException e) { Review Comment: Yes to both, in e16486a7: a runtime failure of `getChangedPaths` is treated as unknown changes (logged at warning level, the branch moves to the new head with an empty state, so it is not repeated on every call), and a runtime failure of `getBranchHeadSha` or `getLatestCommitShaAtCommit` is wrapped into `FlowRegistryException`. Covered by `testRuntimeFailureOfComparisonDropsEverythingAndMovesToNewHead`, `testRuntimeFailureOfHeadCheckIsRegistryFailure` and `runtimeFailureOfLatestCommitAtCommitIsRegistryFailure`. ########## nifi-extension-bundles/nifi-extension-utils/nifi-git-flow-registry/src/test/java/org/apache/nifi/registry/flow/git/AbstractGitFlowRegistryClientTest.java: ########## @@ -161,10 +174,85 @@ void createBranchUnsupportedThrowsFlowRegistryException() { assertTrue(repositoryClient.getCreatedBranchCommit().isEmpty()); } + @Test + void repeatedQuestionsForUnchangedBranchAreAnsweredFromCache() throws Exception { + final GitRepositoryClient repositoryClient = mockRepositoryClient(new AtomicReference<>("head-1")); + when(repositoryClient.getLatestCommitShaAtCommit(FLOW_PATH, "head-1")).thenReturn(Optional.of("commit-1")); + when(repositoryClient.getContentFromCommit(FLOW_PATH, "head-1")).thenAnswer(invocation -> getFlowContent("Flow 1")); + + final TestGitFlowRegistryClient flowRegistryClient = createFlowRegistryClient(repositoryClient); + final FlowRegistryClientConfigurationContext context = createContext("main", "[.].*"); + + for (int i = 0; i < 3; i++) { + assertEquals(Optional.of("commit-1"), flowRegistryClient.getLatestVersion(context, FLOW_LOCATION)); + final RegisteredFlow flow = flowRegistryClient.getFlow(context, FLOW_LOCATION); + assertEquals("Flow 1", flow.getName()); + assertEquals("bucket-a", flow.getBucketIdentifier()); + assertEquals("main", flow.getBranch()); + } + + verify(repositoryClient, times(6)).getBranchHeadSha("main"); + verify(repositoryClient, times(1)).getLatestCommitShaAtCommit(FLOW_PATH, "head-1"); + verify(repositoryClient, times(1)).getContentFromCommit(FLOW_PATH, "head-1"); + verify(repositoryClient, never()).getLatestCommitSha(anyString(), anyString()); + verify(repositoryClient, never()).getContentFromBranch(anyString(), anyString()); + } + + @Test + void writeIsVisibleOnNextRead() throws Exception { Review Comment: Done in e16486a7, as `commitIsVisibleOnNextRead`: it reads the latest version and the flow at the first head, commits a new snapshot through `registerFlowSnapshot`, then checks that both are read again at the new head (the flow content is read once at each head). Keeping cached flows for changed paths makes it fail. The earlier test registered a new flow, which cannot be read before the write. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
