markap14 commented on code in PR #11872: URL: https://github.com/apache/nifi/pull/11872#discussion_r4592090113
########## nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FlowEvolutionStatus.java: ########## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +import java.util.Collection; + +/** + * Reports local progress through the evolve-drain phase of a pending Connector flow evolution. Applying the upgraded flow + * after the evolve-drain phase is tracked through {@link ConnectorState}, as for a configuration update. + */ +public enum FlowEvolutionStatus { + /** + * No evolution is pending, so the node no longer needs the outgoing flow. + */ + INACTIVE, + + /** + * The node still has the outgoing flow and has not finished draining it. + */ + EVOLVE_DRAIN_PENDING, + + /** + * The node has drained the outgoing flow and has not yet applied the upgraded flow. + */ + EVOLVE_DRAIN_COMPLETE, Review Comment: I presume the intent is for the lifecycle to go from EVOLVE_DRAIN_PENDING -> EVOLVE_DRAIN_COMPLETE -> INACTIVE ? It's not really clear. From the comments here and looking at the enum it seems like EVOLVE_DRAIN_COMPLETE should be followed by EVOLUTION_COMPLETE or something but there is no such thing. Can we clarify the expected lifecycle in the javadocs? ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -1041,6 +1097,19 @@ public Future<Void> stop(final FlowEngine scheduler) { return stopCompleteFuture; } + private boolean prepareFlowEvolutionStop(final CompletableFuture<Void> stopCompleteFuture) { Review Comment: This method is confusing. It's a `prepare` method. But it's unclear what the return value is really telling us. And I would not expect a method named `prepareFlowEvolutionStop` to then call `abortFlowEvolution` - i would expect it only to do prep work ########## nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java: ########## @@ -976,6 +976,16 @@ default CompletableFuture<Void> stopComponents() { */ void restoreFlowPreservingIdentifiers(VersionedExternalFlow proposedSnapshot); + /** + * Restores a previously persisted flow while controlling whether components resume their captured scheduled state. + * When {@code resumeComponentState} is false, the flow structure is restored so queued FlowFiles can reattach, but + * no component is started or enabled. + * + * @param proposedSnapshot the proposed flow whose Instance Identifiers should be used as runtime identifiers + * @param resumeComponentState whether components should resume the running or enabled state captured in the snapshot + */ + void restoreFlowPreservingIdentifiers(VersionedExternalFlow proposedSnapshot, boolean resumeComponentState); Review Comment: This is confusing to me. The method is named as if it will restore the flow without change IDs (which would be presumed). But the javadocs makes it sound like the intent is to restore the flow while optionally resuming the components' states. I guess this is intended just as an override to the `restoreFlowPreservingIdentifiers` method that takes just the VersionedExternalFlow argument, but it's confusing here, given the name and the JavaDocs. I also wonder if we need the override, rather than just adding an argument to the existing method. It's also worth noting that we tend to prefer enums on public methods and especially interfaces, rather than `boolean`s. My recommendation, I think, would be to just add a new enum argument to the existing method and eliminate this one. Something like: ``` void restoreFlowPreservingIdentifiers(VerisonedExternalFlow proposedSnapshot, ResumeComponentState resumeStates) ``` Where `ResumeComponentState` has two values, `RESUME_STATES`, `LEAVE_STOPPED` or `RESUME_STATES`, `LEAVE_UNCHANGED` or whatever. This helps to make it very clear when passing the argument exactly what your intent is. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -766,6 +815,11 @@ private void releaseWorkingFlowContext(final WorkingFlowContextState workingCont @Override public void abortUpdate(final Throwable cause) { + if (isFlowEvolutionInProgress() || hasFlowEvolutionFailed()) { + failFlowEvolution(cause == null ? null : cause.getMessage(), cause); + return; + } + stateTransition.setCurrentState(ConnectorState.UPDATE_FAILED); Review Comment: [Claude Opus 5.5] Moving to UPDATE_FAILED here leaves any pending stop futures incomplete forever. `StandardConnectorStateTransition` completes stop futures only when the state becomes STOPPED. The latest push fixes the cancel race that led here, but there's still at least one path that reaches this point with a stop pending. A STOP that arrives while the upgraded flow is being applied (PREPARING_FOR_UPDATE, UPDATING, or UPDATED) gets here. The server [accepts STOP from any state](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L3812), even though the UI doesn't offer it in those states. 1. `stop()` sets STOPPING. 2. The coordinator's `prepareForUpdate` or `waitForState` then fails, and this method sets UPDATE_FAILED. 3. If that happens before `stopComponent` runs, `stopComponent` [returns early](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1279-L1281) without completing its future. Separately, `Connector.stop` and `Connector.applyUpdate` can end up running at the same time on different threads. Anything that waits on a stop then blocks forever. [Orphan Connector removal](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java#L1117) waits with no limit while a reconnecting node holds the flow service write lock. `restartConnector` waits with no limit too, and so would any future caller. Rather than fix each race on its own, I'd make `StandardConnectorStateTransition` guarantee that every stop future completes: - When the state moves to anything other than STOPPING or STOPPED, complete the pending stop futures *exceptionally*, with a message naming the new state. - Have `stopComponent` complete its own future the same way before it returns early. Completing them exceptionally matters, because callers like orphan removal go on to purge, which requires STOPPED. The normal configuration-update path has the same gap today, so this fixes both. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -1619,6 +1775,691 @@ public boolean isExtensionMissing() { return extensionMissing; } + @Override + public boolean isPendingFlowEvolution() { + synchronized (flowEvolutionLock) { + if (extensionMissing) { + return false; + } + + final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot; + final Bundle snapshotBundle = snapshot == null ? null : snapshot.bundle(); + return snapshotBundle != null && !sameBundle(snapshotBundle, bundleCoordinate); + } + } + + @Override + public FlowEvolutionStatus getReportedFlowEvolutionStatus() { + return flowEvolutionProgress.get().reportedStatus(); + } + + private boolean isEvolveDrainComplete() { + return flowEvolutionProgress.get().isDrainComplete(); + } + + private boolean isApplyingEvolvedFlow() { + return getReportedFlowEvolutionStatus() == FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE; + } + + boolean hasFlowEvolutionFailed() { + return flowEvolutionProgress.get().hasFailed(); + } + + private VersionedProcessGroup getAuthoritativeManagedProcessGroup() { + final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot; + return snapshot == null ? null : snapshot.flow(); + } + + @Override + public ConnectorFlowSnapshot getAuthoritativeFlowSnapshot() { + return authoritativeFlowSnapshot; + } + + @Override + public boolean restorePendingFlowEvolution(final VersionedConnector persisted, final List<VersionedConfigurationStep> activeConfiguration, + final List<VersionedConfigurationStep> workingConfiguration) throws FlowUpdateException { + final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted); + final VersionedProcessGroup persistedManagedGroup = persisted.getManagedProcessGroup(); + final Map<String, SecretReference> parameterSecretReferences = ParameterSecretReferences.convertManagedParameterSecretReferences(persisted); + // Components in the persisted group may name NAR versions that are no longer installed. A missing coordinate + // is rewritten only when exactly one compatible bundle is installed. A ghost has no initialization context, + // and its coordinates have to stay as persisted. + if (!extensionMissing) { + resolvePersistedBundles(persistedManagedGroup); + } + + if (extensionMissing && persistedManagedGroup != null) { + // Adopt before inheritConfiguration. A flow save with no snapshot omits the managed group, and inherit + // rebuilds the live group before the persisted one is installed below. + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + return false; + } + + if (!extensionMissing && !sameManagedBundle(managedFlowBundle) && persistedManagedGroup != null) { + // Skip inheritConfiguration. Its applyUpdate would replace the pre-upgrade group with the new bundle's + // flow. The restored group keeps the previous bundle, so the next start evolves it. + restorePersistedFlow(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, true); Review Comment: [Claude Opus 5.5] A failure here can permanently lose the pre-upgrade flow and the Connector's configuration. Here's how: 1. At startup, `restorePersistedFlow` resolves secrets before it touches the flow. If the secrets provider can't be reached, the parameter provider throws an `IllegalStateException` ([StandardParameterProviderNode#L295](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java#L295)), and nothing catches it before `syncConnector`. 2. Because the restore comes first, `loadConfigurationsWithoutRebuildingFlow` and `adoptAuthoritativeSnapshot` never run. The Connector holds no snapshot and no configuration, so it isn't even treated as pending. 3. `syncConnector` calls `markInvalid`, but the next validation clears that. 4. On the next flow save, from any change anywhere, `mapConnector` writes no managed group, parameter context, or managed-flow bundle, because there's no snapshot ([#L1151](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1151)). It also writes empty active and working configurations, taken from the live Connector ([#L1132](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1132), [#L1138](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1138)). 5. After that, a restart with the vault back up finds nothing to restore. The queued FlowFiles stay in the repository as orphans with no connection to rejoin. The extension-missing path in `restorePersistedManagedFlow` already handles this case ([#L1866-L1873](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1866-L1873)). I'd do the same here: 1. Load the configurations before restoring the flow. Loading doesn't depend on the managed flow, and doing it first means the snapshot below records the real active configuration, since `adoptAuthoritativeSnapshot` copies it from the live Connector ([#L1966](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1966)). 2. Catch a failed restore and call `markManagedFlowRestoreFailed()`. `verifyCanStart` and the START action already block on that, with a message telling the user to restart. 3. Record the persisted snapshot anyway, and return `true` so that `syncConnector` doesn't rebuild the flow from the new bundle. It's also worth considering whether an unreachable secrets provider should count as an unresolved secret. The code already handles that case by leaving components stopped. For a test, an existing restore test in `TestStandardConnectorNode` could add a secrets manager that throws. It should check that START is refused and that the mapped Connector still has its managed group and active configuration. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java: ########## @@ -586,10 +618,155 @@ private void syncFromProviderForRead(final ConnectorNode connector) { } @Override - public Future<Void> startConnector(final ConnectorNode connector) { + public Future<Void> startConnector(final ConnectorNode connector, final ConnectorUpdateContext persistenceContext) { + Objects.requireNonNull(persistenceContext, "Flow persistence context is required"); + + final String connectorId = connector.getIdentifier(); + final CompletableFuture<Void> runningEvolution = evolveThenStartFuturesByConnectorId.get(connectorId); + if (runningEvolution != null) { + return runningEvolution; + } + if (!connector.isPendingFlowEvolution()) { + return connector.start(lifecycleExecutor); + } + + final CompletableFuture<Void> evolutionFuture = new CompletableFuture<>(); + final CompletableFuture<Void> existingEvolution = evolveThenStartFuturesByConnectorId.putIfAbsent(connectorId, evolutionFuture); + if (existingEvolution != null) { + return existingEvolution; + } + + try { + connector.prepareFlowEvolution(); Review Comment: [Claude Opus 5.5] START on a pending Connector fails silently when the Connector isn't STOPPED or UPDATE_FAILED: - `verifyCanStart` doesn't check the state, so the replicated START passes verification on every node. - `prepareFlowEvolution` then throws on a node that is STOPPING, PURGING, or running a user-requested drain. A user drain is allowed while an upgrade is pending, and STOPPING happens, for example, while a node is still retrying the stop of the pre-upgrade flow. - The catch below puts the error on the future without logging it, and `StandardConnectorDAO.startConnector` ignores the future, so the user gets a success response. - In a cluster, that node keeps reporting `EVOLVE_DRAIN_PENDING`, so every node where START worked waits on it in `waitForFlowEvolutionStatus` with no limit. I'd reject this START during verification when an upgrade is pending and the state isn't STOPPED or UPDATE_FAILED, and log the failure here as well. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorClusterUnavailableException.java: ########## @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.components.connector; + +/** + * Thrown when a clustered Connector request cannot be completed because no node is connected yet. + */ +public class ConnectorClusterUnavailableException extends RuntimeException { Review Comment: [Claude Opus 5.5] Minor: the replicator methods already declare `IOException`. Extending `IOException` here would make this a checked exception, which matches how we usually signal an expected condition. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -3051,6 +3909,57 @@ private List<DescribedValue> fetchAllowableValues(final String stepName, final S return allowableValues; } + /** + * Progress through a pending flow evolution. The drain status is kept when the evolution fails, so a retry knows whether + * this node has already drained the outgoing flow. + * + * @param drainStatus {@link FlowEvolutionStatus#INACTIVE} when no evolution is pending, otherwise this node's evolve-drain progress + * @param failureExplanation why the evolution failed, or null when it has not failed + */ + private record FlowEvolutionProgress(FlowEvolutionStatus drainStatus, String failureExplanation) { + private static final FlowEvolutionProgress NONE = new FlowEvolutionProgress(FlowEvolutionStatus.INACTIVE, null); + + FlowEvolutionStatus reportedStatus() { + return hasFailed() ? FlowEvolutionStatus.FAILED : drainStatus; + } + + boolean isDrainComplete() { + return drainStatus == FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE; + } + + boolean hasFailed() { + return failureExplanation != null; + } + + FlowEvolutionProgress withDrainStatus(final FlowEvolutionStatus status) { + return new FlowEvolutionProgress(status, failureExplanation); + } + + FlowEvolutionProgress replaceDrainStatus(final FlowEvolutionStatus expected, final FlowEvolutionStatus replacement) { + return drainStatus == expected ? withDrainStatus(replacement) : this; + } + + FlowEvolutionProgress withFailureIfAbsent(final String explanation) { + return hasFailed() ? this : new FlowEvolutionProgress(drainStatus, explanation); + } + } + + /** + * One attempt at a pending flow evolution. The futures are each set at most once per attempt, and are read and written + * only while holding the evolution lock, so closing the attempt cancels this attempt's work and never a newer attempt's. + */ + private static final class FlowEvolutionAttempt { Review Comment: [Claude Opus 5.5] Not a blocker, but worth considering: this PR adds a little over 900 lines to `StandardConnectorNode`, which is now about 4,000 lines. Most of the new code is flow evolution: - this class and `FlowEvolutionProgress`; - `flowEvolutionLock`; - the prepare, evolve-drain, begin-update, fail, and retry-stop methods; - the snapshot handling. Pulling that into its own class, owned by the node and called by it, would put all of the evolution state in one place. That would make it much easier to see which thread changes what, and to unit test the evolution states directly. The node would keep its lifecycle methods and delegate the evolution parts. Curious whether you think that's worthwhile. ########## nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorChangeVersionDrainIT.java: ########## @@ -0,0 +1,481 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.components.connector.FlowEvolutionStatus; +import org.apache.nifi.flow.VersionedConnector; +import org.apache.nifi.flow.VersionedConnectorState; +import org.apache.nifi.flow.VersionedConnectorValueReference; +import org.apache.nifi.flow.VersionedParameter; +import org.apache.nifi.security.encryption.PropertyEncryptionEncoder; +import org.apache.nifi.tests.system.FlowDefinitionFiles; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO; +import org.apache.nifi.web.api.dto.StateEntryDTO; +import org.apache.nifi.web.api.entity.ComponentStateEntity; +import org.apache.nifi.web.api.entity.ConnectionEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ParameterProviderEntity; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Single-node upgrade-drain scenarios for a version change. + */ +public class ConnectorChangeVersionDrainIT extends AbstractChangeVersionDrainIT { + + private static final String UPGRADE_SECRET = "Upgrade Secret"; + private static final String PROVIDER_NAME = "PropertiesParameterProvider"; + private static final String SECRET_NAME = "upgrade-secret"; + private static final String SECRET_FULLY_QUALIFIED_NAME = PROVIDER_NAME + ".Parameters." + SECRET_NAME; + private static final String GENERATOR = "GenerateFlowFile"; + private static final String CANCELLED_EVOLUTION = "Flow evolution was cancelled"; + private static final String REWIRED_ENDPOINT = "Rewired Endpoint"; + + @Test + @Timeout(360) + public void testPreUpgradeParameterSecretIsRefreshedAndCanBeRestored() throws Exception { + final SecretConnector secretConnector = createSecretConnector("original-secret"); + changeVersion(secretConnector.connectorId(), DRAIN_V2); + getClientUtil().updateParameterProviderProperties(secretConnector.parameterProvider(), Map.of("parameters", SECRET_NAME + "=rotated-secret")); + + getNiFiInstance().stop(); Review Comment: [Claude Opus 5.5] Only four tests in this class actually check behavior across a restart: - `testPreUpgradeParameterSecretIsRefreshedAndCanBeRestored` - `testPendingUpgradeEditsThenStartsWithoutReEvolving`, which restarts twice - `testRestartDuringDrainingComesBackPending` - `testPendingUpgradeBlocksEndingTroubleshootingWithQueuedFlowFiles` Could these become one test with a single restart? Before restarting, set up one connector per scenario: - one with a pending upgrade, working-configuration edits and a secret; - one in troubleshooting with an upgrade pending; - one draining, with its own `Gate File Path` so it can be held and released separately from the others. Then restart once and check each connector. The second restart in `testPendingUpgradeEditsThenStartsWithoutReEvolving` checks the saved flow after the upgrade finishes. It can become a read of the flow file with NiFi still running. Together with the change I suggested for `stopAndAssertNotPending`, this takes the class from 14 restarts to 1. ########## nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/AbstractConnectorUpgradeDrainIT.java: ########## @@ -0,0 +1,406 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.tests.system.connectors; + +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.flow.Bundle; +import org.apache.nifi.flow.VersionedConfigurationStep; +import org.apache.nifi.flow.VersionedConnector; +import org.apache.nifi.flow.VersionedProcessGroup; +import org.apache.nifi.flow.VersionedProcessor; +import org.apache.nifi.nar.NarState; +import org.apache.nifi.tests.system.AbstractNarSwapMigrationIT; +import org.apache.nifi.tests.system.FlowDefinitionFiles; +import org.apache.nifi.tests.system.NiFiInstance; +import org.apache.nifi.tests.system.nar.NarUploadUtil; +import org.apache.nifi.toolkit.client.ConnectorClient; +import org.apache.nifi.toolkit.client.NiFiClientException; +import org.apache.nifi.web.api.dto.BundleDTO; +import org.apache.nifi.web.api.dto.ConnectorDTO; +import org.apache.nifi.web.api.dto.NarSummaryDTO; +import org.apache.nifi.web.api.entity.ConnectionEntity; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity; +import org.apache.nifi.web.api.entity.ProcessorEntity; +import org.junit.jupiter.api.BeforeEach; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Upgrade-drain scenarios that hold for every way a Connector can move to a new version. Each subclass supplies the + * trigger: a NAR swap followed by a restart, or a version change while the Connector is stopped. + */ +public abstract class AbstractConnectorUpgradeDrainIT extends AbstractNarSwapMigrationIT { + + private static final String GATE_FILE = "upgrade-drain-gate.txt"; + static final String PREPARATION_FAILED = "flow evolution preparation failed"; + static final String REMOVED_CONNECTION = "Removed Connection"; + static final String REMOVED_PROCESSOR_CHAIN = "Removed Processor Chain"; + static final String RELAY_PROCESSOR = "RelayFlowFile"; + static final String REMOVED_PROCESSOR = "RemovedFlowFile"; + static final String SWAPPED_PROCESSOR = "SwappedFlowFile"; + + static final String DRAIN_CONNECTOR_TYPE = "org.apache.nifi.connectors.tests.system.UpgradeDrainConnector"; + static final String DRAIN_BUNDLE_GROUP = "org.apache.nifi"; + static final String DRAIN_BUNDLE_ARTIFACT = "nifi-connector-upgrade-drain-nar"; + static final String DRAIN_V1 = "1.0.0"; + static final String DRAIN_V2 = "2.0.0"; + private static final String DRAIN_NARS_LOCATION = "target/nifi-connector-upgrade-drain-nars"; + private static final Map<String, String> DRAIN_NAR_IDS = Map.of( + DRAIN_V1, "nifi-connector-upgrade-drain-v1-nar", + DRAIN_V2, "nifi-connector-upgrade-drain-v2-nar"); + + /** + * Makes the version 2 UpgradeDrainConnector misbehave. The Connector reads the named file from its node's working + * directory, and the file contents select the behavior. Faults that share a file replace each other. + */ + protected enum UpgradeDrainFault { + PREPARATION_RETURNS_NULL_FUTURE("upgrade-drain-preparation-result.txt", "NULL_FUTURE"), + PREPARATION_FAILS("upgrade-drain-preparation-result.txt", "FAILED_FUTURE"), + PREPARATION_UPDATES_PRE_UPGRADE_FLOW("upgrade-drain-preparation-result.txt", "UPDATE_PRE_UPGRADE_FLOW"), + ACTIVE_FLOW_RETURNS_NULL("upgrade-drain-active-flow.txt", "NULL"), + ACTIVE_FLOW_THROWS("upgrade-drain-active-flow.txt", "THROW"), + APPLY_FAILS("upgrade-drain-apply-fail.txt", ""); + + private final String fileName; + private final String contents; + + UpgradeDrainFault(final String fileName, final String contents) { + this.fileName = fileName; + this.contents = contents; + } + } + + /** + * Identifies a Connector bundle in assertions. A null version matches any version of the artifact. + */ + protected record ExpectedBundle(String artifact, String version) { + + String recordPrefix() { + return "org.apache.nifi:" + artifact + ":" + (version == null ? "" : version); + } + + void assertMatches(final Bundle bundle, final String description) { + assertNotNull(bundle, description); + assertTrue(bundle.getArtifact().contains(artifact), description + " " + bundle.getArtifact()); + if (version != null) { + assertEquals(version, bundle.getVersion(), description); + } + } + } + + /** + * Creates the Connector on its outgoing version and applies the given configuration. + */ + protected abstract String createOutgoingConnector(Map<String, String> properties) throws Exception; + + /** + * Moves the stopped Connector to its incoming version, leaving it STOPPED with the upgrade pending. + */ + protected abstract void triggerUpgrade(String connectorId) throws Exception; + + protected abstract ExpectedBundle outgoingBundle(); + + protected abstract ExpectedBundle incomingBundle(); + + @BeforeEach + public void clearFaults() { + for (final UpgradeDrainFault fault : UpgradeDrainFault.values()) { + clearFault(fault); + } + } + + protected void injectFault(final UpgradeDrainFault fault) throws IOException { + for (int nodeIndex = 1; nodeIndex <= nodeCount(); nodeIndex++) { + injectFault(nodeIndex, fault); + } + } + + protected void injectFault(final int nodeIndex, final UpgradeDrainFault fault) throws IOException { + writeToNodeDirectory(nodeIndex, fault.fileName, fault.contents); + } + + protected void clearFault(final UpgradeDrainFault fault) { + deleteFromNodeDirectories(fault.fileName); + } + + protected String createOutgoingConnector(final String mode, final boolean removeConnection, final String comment) throws Exception { + return createOutgoingConnector(properties(mode, removeConnection, comment)); + } + + protected String queueStoppedUpgrade(final String mode, final boolean removeConnection, final String comment) throws Exception { + return queueAtLeastOneFlowFileAndStop(createOutgoingConnector(mode, removeConnection, comment)); + } + + protected String queueAtLeastOneFlowFileAndStop(final String connectorId) throws Exception { + holdFlowFiles(); + getClientUtil().startConnector(connectorId); + waitForConnectorMinQueueCount(connectorId, 1); + getClientUtil().stopConnector(connectorId); + return connectorId; + } + + protected void restartKeepingIncomingVersion() { + getNiFiInstance().stop(); + getNiFiInstance().start(true); + waitForCluster(); + } + + protected void uploadDrainNars(final String... versions) throws Exception { + final File narsLocation = new File(DRAIN_NARS_LOCATION); + if (!narsLocation.exists()) { + throw new IllegalStateException("NARs location does not exist at: " + narsLocation.getAbsolutePath()); + } + + final NarUploadUtil narUploadUtil = new NarUploadUtil(getNifiClient()); + for (final String version : versions) { + final NarSummaryDTO nar = narUploadUtil.uploadNar(narsLocation, DRAIN_NAR_IDS.get(version)); + waitFor(narUploadUtil.getWaitForNarStateSupplier(nar.getIdentifier(), NarState.INSTALLED)); + } + } + + protected String createDrainConnector(final String version, final Map<String, String> properties) throws NiFiClientException, IOException, InterruptedException { + final ConnectorEntity connector = getClientUtil().createConnector(DRAIN_CONNECTOR_TYPE, DRAIN_BUNDLE_GROUP, DRAIN_BUNDLE_ARTIFACT, version); + getClientUtil().configureConnector(connector, "Upgrade", properties); + getClientUtil().applyConnectorUpdate(connector); + getClientUtil().waitForValidConnector(connector.getId()); + return connector.getId(); + } + + /** + * Sends the version change the way the UI does: a PUT carrying only the new bundle. + */ + protected ConnectorEntity changeVersion(final String connectorId, final String version) throws NiFiClientException, IOException { + final ConnectorClient connectorClient = getNifiClient().getConnectorClient(); + final ConnectorEntity current = connectorClient.getConnector(connectorId); + final ConnectorDTO component = new ConnectorDTO(); + component.setId(connectorId); + component.setBundle(new BundleDTO(DRAIN_BUNDLE_GROUP, DRAIN_BUNDLE_ARTIFACT, version)); + + final ConnectorEntity update = new ConnectorEntity(); + update.setId(connectorId); + update.setRevision(current.getRevision()); + update.setComponent(component); + return connectorClient.updateConnector(update); + } + + /** + * Stops the generator and lets the terminator drop queued FlowFiles on every node. + */ + protected void releaseFlowFiles() throws IOException { + for (int nodeIndex = 1; nodeIndex <= nodeCount(); nodeIndex++) { + releaseFlowFiles(nodeIndex); + } + } + + /** + * Stops the generator and lets the terminator drop queued FlowFiles on one node. + */ + protected void releaseFlowFiles(final int nodeIndex) throws IOException { + writeToNodeDirectory(nodeIndex, GATE_FILE, ""); + } + + /** + * Lets the generator run and makes the terminator hold every FlowFile on every node. + */ + protected void holdFlowFiles() { + deleteFromNodeDirectories(GATE_FILE); + } + + private void writeToNodeDirectory(final int nodeIndex, final String fileName, final String contents) throws IOException { + Files.writeString(new File(nodeDirectory(nodeIndex), fileName).toPath(), contents); + } + + private void deleteFromNodeDirectories(final String fileName) { + for (final File directory : instanceDirectories()) { + final File file = new File(directory, fileName); + if (file.exists()) { + assertTrue(file.delete(), file.getAbsolutePath()); + } + } + } + + protected void stopAndAssertNotPending(final String connectorId) throws IOException { Review Comment: [Claude Opus 5.5] The new system tests restart NiFi far too often, and this helper is the biggest reason. It stops NiFi only so the test can read `flow.json.gz`. In the classes that share one NiFi instance between tests (`ConnectorChangeVersionDrainIT` and `ClusteredConnectorChangeVersionDrainIT`), the `@AfterEach` in `AbstractNarSwapMigrationIT` then sees that NiFi is not running and starts it again. So every test that ends this way pays for a full restart, even though it never checks anything across a restart. Here is the count across the PR: - `ConnectorChangeVersionDrainIT`: 11 of its 13 tests restart NiFi, for 14 restarts in total. - `ClusteredConnectorChangeVersionDrainIT`: 1 restart of the 2-node cluster, through this helper. - `ClusteredConnectorUpgradeDrainIT` and `ConnectorUpgradeDrainProviderIT`: each builds a brand-new environment and then restarts it once for the NAR swap. - `ConnectorParameterContextIT`: 1 added restart. About 10 of those restarts happen only to read the flow file at the end of a test. The flow is saved shortly after each change, and `assertPersistedOutgoing` already reads the file while NiFi is running. Can this helper, and the `getNiFiInstance().stop()` calls followed by `FlowDefinitionFiles.readConnector` at the end of tests in `ConnectorChangeVersionDrainIT`, read the file with NiFi still running? Re-reading it inside a `waitFor` until it shows the expected bundle would handle the short delay before the save. Where the REST API already shows the same thing (for example, a flow evolution status of INACTIVE), checking through REST is simpler still. Overall, I'd like the new tests to get down to two that restart NiFi: one test that checks what survives a restart (see my comment in `ConnectorChangeVersionDrainIT`), and the clustered NAR-swap startup test in `ClusteredConnectorUpgradeDrainIT`. ########## nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorUpgradeDrainProviderIT.java: ########## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.tests.system.connectors; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; +import org.apache.nifi.components.connector.ConnectorState; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO; +import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO; +import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO; +import org.apache.nifi.web.api.entity.ConnectorEntity; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; + +/** + * A NAR-swap upgrade applies configuration supplied by the external provider. + */ +public class ConnectorUpgradeDrainProviderIT extends AbstractNarSwapUpgradeDrainIT { + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + private static final String COMMENT = "Comment"; + private static final String LEGACY_NOTE = "Legacy Note"; + private static final String DRAIN_LABEL = "Drain Label"; + private static final String SUPPLIED_COMMENT = "supplied-by-provider"; + + private final Path providerState = createProviderState(); + + @Override + protected Map<String, String> getNifiPropertiesOverrides() { + return Map.of(NiFiProperties.CONNECTOR_CONFIGURATION_PROVIDER_IMPLEMENTATION, "org.apache.nifi.connectors.tests.system.FileConnectorConfigurationProvider", NiFiProperties + .CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX + "state.directory", providerState.toAbsolutePath().toString()); + } + + @Test + @Timeout(240) + public void testProviderSuppliedConfigurationIsAppliedByUpgrade() throws Exception { + final Map<String, String> initial = properties("DRAIN", true, "shared"); + initial.put(LEGACY_NOTE, "v1-only"); + final String connectorId = createConnector("UpgradeDrainConnector", initial); + + getNiFiInstance().stop(); Review Comment: [Claude Opus 5.5] This test needs its own environment because of the provider properties, and the NAR swap adds a restart on top of that. Is the NAR swap important to what this test checks? If the provider-supplied configuration gets applied the same way when the version is changed through the REST API (as in `AbstractChangeVersionDrainIT`), this test could use a version change and skip the restart. If the NAR-swap startup path is the point of the test, could it be folded into `ClusteredConnectorUpgradeDrainIT` instead, so that we have one NAR-swap restart test instead of two? ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -1619,6 +1775,691 @@ public boolean isExtensionMissing() { return extensionMissing; } + @Override + public boolean isPendingFlowEvolution() { + synchronized (flowEvolutionLock) { + if (extensionMissing) { + return false; + } + + final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot; + final Bundle snapshotBundle = snapshot == null ? null : snapshot.bundle(); + return snapshotBundle != null && !sameBundle(snapshotBundle, bundleCoordinate); + } + } + + @Override + public FlowEvolutionStatus getReportedFlowEvolutionStatus() { + return flowEvolutionProgress.get().reportedStatus(); + } + + private boolean isEvolveDrainComplete() { + return flowEvolutionProgress.get().isDrainComplete(); + } + + private boolean isApplyingEvolvedFlow() { + return getReportedFlowEvolutionStatus() == FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE; + } + + boolean hasFlowEvolutionFailed() { + return flowEvolutionProgress.get().hasFailed(); + } + + private VersionedProcessGroup getAuthoritativeManagedProcessGroup() { + final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot; + return snapshot == null ? null : snapshot.flow(); + } + + @Override + public ConnectorFlowSnapshot getAuthoritativeFlowSnapshot() { + return authoritativeFlowSnapshot; + } + + @Override + public boolean restorePendingFlowEvolution(final VersionedConnector persisted, final List<VersionedConfigurationStep> activeConfiguration, + final List<VersionedConfigurationStep> workingConfiguration) throws FlowUpdateException { + final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted); + final VersionedProcessGroup persistedManagedGroup = persisted.getManagedProcessGroup(); + final Map<String, SecretReference> parameterSecretReferences = ParameterSecretReferences.convertManagedParameterSecretReferences(persisted); + // Components in the persisted group may name NAR versions that are no longer installed. A missing coordinate + // is rewritten only when exactly one compatible bundle is installed. A ghost has no initialization context, + // and its coordinates have to stay as persisted. + if (!extensionMissing) { + resolvePersistedBundles(persistedManagedGroup); + } + + if (extensionMissing && persistedManagedGroup != null) { + // Adopt before inheritConfiguration. A flow save with no snapshot omits the managed group, and inherit + // rebuilds the live group before the persisted one is installed below. + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + return false; + } + + if (!extensionMissing && !sameManagedBundle(managedFlowBundle) && persistedManagedGroup != null) { + // Skip inheritConfiguration. Its applyUpdate would replace the pre-upgrade group with the new bundle's + // flow. The restored group keeps the previous bundle, so the next start evolves it. + restorePersistedFlow(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, true); + loadConfigurationsWithoutRebuildingFlow(activeConfiguration, workingConfiguration); + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + return true; + } + + return false; + } + + @Override + public void restorePersistedManagedFlow(final VersionedConnector persisted) { + if (!extensionMissing) { + throw new IllegalStateException("Cannot restore the persisted managed flow for " + this + " because its extension is not missing"); + } + + final VersionedProcessGroup persistedManagedGroup = persisted.getManagedProcessGroup(); + if (persistedManagedGroup == null) { + return; + } + + final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted); + final Map<String, SecretReference> parameterSecretReferences = ParameterSecretReferences.convertManagedParameterSecretReferences(persisted); + + // Install the persisted flow while ghosted so queues exist before the FlowFile repository recovers. + // Components stay stopped: a ghost cannot start, and resuming would run them under a missing extension. + try { + restorePersistedFlow(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, false); + } catch (final Exception e) { + logger.error("{} failed to restore the persisted managed flow while ghosted; " + + "queued FlowFiles may not rejoin until NiFi restarts after the Connector becomes loadable again", this, e); + markManagedFlowRestoreFailed(); + } + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + } + + @Override + public void restoreTroubleshootingManagedFlow(final VersionedConnector persisted) throws FlowUpdateException { + final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted); + final VersionedProcessGroup persistedManagedGroup = persisted.getManagedProcessGroup(); + final Map<String, SecretReference> parameterSecretReferences = ParameterSecretReferences.convertManagedParameterSecretReferences(persisted); + + if (extensionMissing && persistedManagedGroup != null) { + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + restoreTroubleshootingFlow(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences); + return; + } + + if (sameManagedBundle(managedFlowBundle) && persistedManagedGroup != null) { + try { + restorePersistedFlow(persistedManagedGroup, null, Map.of(), true); + adoptAuthoritativeSnapshot(persistedManagedGroup, persisted.getManagedParameterContext(), parameterSecretReferences, managedFlowBundle); + } catch (final Exception e) { + logger.error("{} failed to restore the managed flow while restoring Troubleshooting", this, e); + throw new FlowUpdateException("Failed to restore Managed Process Group from persisted flow: " + e.getMessage(), e); + } + } + } + + @Override + public void ensureAuthoritativeFlowSnapshot() { + if (getAuthoritativeManagedProcessGroup() != null || extensionMissing) { + return; + } + + // Without a snapshot, a later NAR upgrade has no pre-upgrade flow to evolve. + verifyCanCaptureAuthoritativeSnapshot(); + final ConnectorFlowSnapshot snapshot = createAuthoritativeSnapshot(); + + synchronized (flowEvolutionLock) { + // Another thread may have installed a snapshot, such as the pre-upgrade flow, while the live flow was being captured + if (getAuthoritativeManagedProcessGroup() != null) { + return; + } + + verifyCanCaptureAuthoritativeSnapshot(); + installAuthoritativeSnapshot(snapshot); + } + } + + private void restoreTroubleshootingFlow(final VersionedProcessGroup persistedManagedGroup, final VersionedParameterContext parameterContext, + final Map<String, SecretReference> parameterSecretReferences) throws FlowUpdateException { + logger.info("{} was persisted in Troubleshooting mode; restoring Managed Process Group from persisted snapshot", this); + try { + restorePersistedFlow(persistedManagedGroup, parameterContext, parameterSecretReferences, false); + } catch (final Exception e) { + logger.error("{} failed to restore the managed flow while restoring Troubleshooting", this, e); + throw new FlowUpdateException("Failed to restore Managed Process Group from persisted flow: " + e.getMessage(), e); + } + } + + /** + * A group saved before managedFlowBundle existed is attributed to the connector bundle stored in the flow definition. + */ + private static Bundle resolveManagedFlowBundle(final VersionedConnector versionedConnector) { + if (versionedConnector.getManagedFlowBundle() != null) { + return versionedConnector.getManagedFlowBundle(); + } + if (versionedConnector.getManagedProcessGroup() != null) { + return versionedConnector.getBundle(); + } + return null; + } + + private boolean sameManagedBundle(final Bundle managedFlowBundle) { + if (managedFlowBundle == null || bundleCoordinate == null) { + return true; + } + return sameBundle(managedFlowBundle, bundleCoordinate); + } + + /** + * Rewrites processor and controller service bundles in a persisted group to the only installed alternative when + * the exact coordinate is missing. A ghost Connector keeps the snapshot exactly as persisted. + */ + void resolvePersistedBundles(final VersionedProcessGroup group) { + if (extensionMissing || group == null || initializationContext == null) { + return; + } + + initializationContext.resolveBundles(group, BundleCompatibility.RESOLVE_BUNDLE); + } + + void adoptAuthoritativeSnapshot(final VersionedProcessGroup flow, final VersionedParameterContext parameterContext, + final Map<String, SecretReference> parameterSecretReferences, final Bundle managedFlowBundle) { + installAuthoritativeSnapshot(new ConnectorFlowSnapshot(flow, parameterContext, parameterSecretReferences, managedFlowBundle, + activeFlowContext.getConfigurationContext().toConnectorConfiguration())); + } + + void restorePersistedFlow(final VersionedProcessGroup flow, final VersionedParameterContext parameterContext, + final Map<String, SecretReference> parameterSecretReferences, final boolean resumeComponents) { + final ParameterSecretReferences.Resolution resolution = resolvePersistedParameterSecrets(parameterContext, parameterSecretReferences); + if (!resolution.undecryptableParameters().isEmpty()) { + logger.warn("{} could not decrypt the persisted values of sensitive parameters {}; components are left stopped", this, resolution.undecryptableParameters()); + } + + if (resumeComponents && unresolvedParameterSecretReferences.isEmpty() && resolution.undecryptableParameters().isEmpty()) { + activeFlowContext.restorePersistedFlow(flow, resolution.parameterContext()); + } else { + activeFlowContext.restorePersistedFlow(flow, resolution.parameterContext(), false); Review Comment: This feels weird to call the same method but in one place rely on 'default values' while in the other passing them explicitly. Probably makes sense to extract the `resumeComponents && unresolvedParameterSecretReferences.isEmpty() && resolution.undecryptableParameters().isEmpty()` into a boolean and just pass that as the last argument, no? ########## nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java: ########## @@ -4112,18 +4112,27 @@ private void verifyCanRemoveMissingComponents(final ProcessGroup processGroup, f final Map<String, VersionedConnection> proposedConnectionsByVersionedId = proposedGroup.getConnections().stream() .collect(Collectors.toMap(VersionedComponent::getIdentifier, Function.identity())); - // match group's current connections to proposed connections to determine if they've been removed + // A missing connection, or one whose source changed, is removed and recreated. Either change drops + // queued data, so reject it before synchronization begins. A destination change updates the connection in place. for (final Connection connection : processGroup.getConnections()) { final String versionedId = connection.getVersionedComponentId().orElse( VersionedComponentFlowMapper.generateVersionedComponentId(connection.getIdentifier())); final VersionedConnection proposedConnection = proposedConnectionsByVersionedId.get(versionedId); + final FlowFileQueue flowFileQueue = connection.getFlowFileQueue(); + if (flowFileQueue.isEmpty()) { + continue; + } + if (proposedConnection == null) { - // connection doesn't exist in proposed connections, make sure it doesn't have any data in it - final FlowFileQueue flowFileQueue = connection.getFlowFileQueue(); - if (!flowFileQueue.isEmpty()) { - throw new IllegalStateException(processGroup + " cannot be updated to the proposed flow because the proposed flow " - + "does not contain a match for " + connection + " and the connection currently has data in the queue."); - } + throw new IllegalStateException(processGroup + " cannot be updated to the proposed flow because the proposed flow " + + "does not contain a match for " + connection + " and the connection currently has data in the queue."); + } + + final String existingSourceId = connection.getSource().getVersionedComponentId().orElse(null); Review Comment: [Claude Opus 5.5] This check applies to every versioned-flow synchronization, including registry version changes, but its four unit tests were removed. Could one combined test come back in `StandardVersionedComponentSynchronizerTest`? It would cover a source change that is rejected while the queue has data and allowed when the queue is empty. Also, the source comparison repeats the one in `removeMissingConnections`. A small shared helper such as `isSourceChanged(Connection, VersionedConnection)` would keep the two from drifting apart. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java: ########## @@ -733,6 +938,74 @@ private void waitForState(final ConnectorNode connector, final Set<ConnectorStat } } + private void waitForFlowEvolutionStatus(final ConnectorNode connector) throws FlowUpdateException, IOException, InterruptedException { Review Comment: [Claude Opus 5.5] This wait still has no time limit. A user can always end it with STOP or cancel drain, because the loop checks for a local failure on every poll. But it never resolves on its own, and the silent-START case on `startConnector` is a concrete way to get here. The UI way out can also be blocked. STOP isn't offered while DRAINING ([#L3425-L3443](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L3425-L3443)), and cancel drain is rejected if any node is STOPPING ([#L1209-L1218](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1209-L1218)). A limit equal to the drain timeout, measured from when this node's drain finishes, would be enough. The admin guide line that describes the wait as unlimited would need updating. ########## nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java: ########## @@ -3870,11 +3892,17 @@ public ConnectorEntity scheduleConnector(final Revision revision, final String i final RevisionUpdate<ConnectorDTO> snapshot = revisionManager.updateRevision(claim, user, () -> { switch (state) { - case RUNNING -> connectorDAO.startConnector(id); + case RUNNING -> connectorDAO.startConnector(id, controllerFacade::saveImmediate); case STOPPED -> connectorDAO.stopConnector(id); default -> throw new IllegalArgumentException("Unsupported scheduled state for Connector: " + state); } - controllerFacade.save(); + // Evolution sets the desired state to RUNNING as soon as it is claimed so a joining node starts too. + // Persisting that state before the hook succeeds makes a failed evolution auto-start on restart. + // The repository saves once the upgraded flow is applied, and a failed evolution puts the desired state back. + final ConnectorNode scheduled = connectorDAO.getConnector(id, ConnectorSyncMode.LOCAL_ONLY); + if (scheduled.getCurrentState() != ConnectorState.DRAINING || !scheduled.isPendingFlowEvolution()) { Review Comment: [Claude Opus 5.5] I think automatically resuming the upgrade after a restart is the right behavior. Today it only happens sometimes, though: - Flow saves record the Connector's desired state, which is RUNNING during the drain. - This save is skipped, but any other save during the drain writes RUNNING. That could be an edit anywhere in the flow, or the pending save that is flushed at shutdown. - So whether a restart resumes the upgrade depends on unrelated activity. If resuming is what we want, I'd drop this special case so that the RUNNING state is always saved when the upgrade begins. The developer guide sentence "The failure is not saved, so after a restart the Connector comes back in the state that was last saved" would need to match. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java: ########## @@ -586,10 +618,155 @@ private void syncFromProviderForRead(final ConnectorNode connector) { } @Override - public Future<Void> startConnector(final ConnectorNode connector) { + public Future<Void> startConnector(final ConnectorNode connector, final ConnectorUpdateContext persistenceContext) { + Objects.requireNonNull(persistenceContext, "Flow persistence context is required"); + + final String connectorId = connector.getIdentifier(); + final CompletableFuture<Void> runningEvolution = evolveThenStartFuturesByConnectorId.get(connectorId); + if (runningEvolution != null) { + return runningEvolution; + } + if (!connector.isPendingFlowEvolution()) { + return connector.start(lifecycleExecutor); + } + + final CompletableFuture<Void> evolutionFuture = new CompletableFuture<>(); + final CompletableFuture<Void> existingEvolution = evolveThenStartFuturesByConnectorId.putIfAbsent(connectorId, evolutionFuture); + if (existingEvolution != null) { + return existingEvolution; + } + + try { + connector.prepareFlowEvolution(); + } catch (final Throwable failure) { + evolutionFuture.completeExceptionally(failure); + evolveThenStartFuturesByConnectorId.remove(connectorId, evolutionFuture); + if (failure instanceof final Error error) { + throw error; + } + return evolutionFuture; + } + + try { + flowEvolutionCoordinatorExecutor.submit(() -> evolveThenStart(connector, evolutionFuture, persistenceContext)); + } catch (final Throwable failure) { + try { + connector.abortUpdate(failure); + } catch (final Throwable cleanupFailure) { + failure.addSuppressed(cleanupFailure); + } + evolutionFuture.completeExceptionally(failure); + evolveThenStartFuturesByConnectorId.remove(connectorId, evolutionFuture); + if (failure instanceof final Error error) { + throw error; + } + } + + return evolutionFuture; + } + + private void evolveThenStart(final ConnectorNode connector, final CompletableFuture<Void> evolutionFuture, final ConnectorUpdateContext persistenceContext) { + boolean startAttempted = false; + try { + final Future<Void> startFuture = runFlowEvolution(connector, persistenceContext); + startAttempted = true; + // The evolution is complete once its start is issued. A later START or STOP then acts on the connector directly. + evolveThenStartFuturesByConnectorId.remove(connector.getIdentifier(), evolutionFuture); + startFuture.get(); + evolutionFuture.complete(null); + } catch (final Throwable failure) { + if (failure instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } + + if (startAttempted) { + logger.error("{} was evolved but did not start", connector, failure); + } else { + logger.error("Flow evolution failed for {}", connector, failure); + } + + handleFlowEvolutionFailure(connector, evolutionFuture, failure, startAttempted); + + if (failure instanceof final Error error) { + throw error; + } + } finally { + evolveThenStartFuturesByConnectorId.remove(connector.getIdentifier(), evolutionFuture); + } + } + + /** + * Runs the evolve-drain phase of the evolution, then applies the upgraded flow as a configuration update. + * + * @return the future for starting the evolved connector + */ + private Future<Void> runFlowEvolution(final ConnectorNode connector, final ConnectorUpdateContext persistenceContext) throws Throwable { + awaitFlowEvolutionDrain(connector, connector.evolveFlow(flowEvolutionCoordinatorExecutor)); + + // No node applies the upgraded flow until every node that still has the outgoing flow has drained it. + waitForFlowEvolutionStatus(connector); + + // Every node has drained, so as at startup the preparation is local and no cluster-wide wait for UPDATING is needed. + // A peer that has not yet observed the drain barrier may still be DRAINING or preparing. + connector.beginEvolvedFlowUpdate(); + connector.prepareForUpdate(); + completeUpdate(connector, persistenceContext, Set.of(ConnectorState.PREPARING_FOR_UPDATE, ConnectorState.UPDATING, ConnectorState.DRAINING)); + + // The connector stayed desired RUNNING throughout the evolution, so the flow saved after the apply already records it as running. + logger.info("{} has been successfully evolved; starting", connector); return connector.start(lifecycleExecutor); } + private void awaitFlowEvolutionDrain(final ConnectorNode connector, final Future<Void> future) throws Throwable { + try { + future.get(flowEvolutionDrainTimeout.toNanos(), TimeUnit.NANOSECONDS); + } catch (final TimeoutException e) { + final FlowUpdateException timedOut = new FlowUpdateException("Flow evolution drain timed out for " + connector, e); + connector.abortUpdate(timedOut); + throw timedOut; + } catch (final InterruptedException e) { + connector.abortUpdate(e); + throw e; + } catch (final CancellationException e) { + throw new FlowUpdateException("Flow evolution was cancelled for " + connector, e); + } catch (final ExecutionException e) { + final Throwable cause = e.getCause() == null ? e : e.getCause(); + if (cause instanceof final CancellationException cancelled) { + throw new FlowUpdateException("Flow evolution was cancelled for " + connector, cancelled); + } + throw cause; + } + } + + private static Duration canonicalFlowEvolutionDrainTimeout(final Duration timeout) { + if (timeout == null || timeout.isZero() || timeout.isNegative()) { + return DEFAULT_FLOW_EVOLUTION_DRAIN_TIMEOUT; + } + + return timeout; + } + + private void handleFlowEvolutionFailure(final ConnectorNode connector, final CompletableFuture<Void> evolutionFuture, + final Throwable failure, final boolean startAttempted) { + // A failed evolution is not saved; the persisted flow stays as it was last saved until the connector is corrected. + // A failed start of the evolved connector is an ordinary failed start, because the upgraded flow has already been saved. + if (!startAttempted) { + try { + // While DRAINING this fails the evolution. Once preparation for the upgraded flow has begun it leaves UPDATE_FAILED. + connector.abortUpdate(failure); + if (connector.getCurrentState() == ConnectorState.STOPPING && connector instanceof final StandardConnectorNode standardConnector) { Review Comment: [Claude Opus 5.5] Minor: the cast to `StandardConnectorNode` to call `retryPreUpgradeStop` suggests this retry belongs inside the node. ########## nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java: ########## @@ -979,7 +1032,7 @@ private void start(final FlowEngine scheduler, final CompletableFuture<Void> sta stateTransition.addPendingStartFuture(startCompleteFuture); return; } - case STOPPED, PREPARING_FOR_UPDATE, UPDATED -> { + case STOPPED, PREPARING_FOR_UPDATE, UPDATED, UPDATE_FAILED -> { Review Comment: [Claude Opus 5.5] Only `StandardConnectorRepository.startConnector` checks for a pending upgrade. `start()` itself doesn't, and at least one caller goes around the repository: - On reconnect, `StandardConnectorAssetSynchronizer` restarts Connectors that are desired RUNNING and whose assets it just downloaded ([#L124-L128](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java#L124-L128)). - A draining Connector is [desired RUNNING](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L2014). - `restartConnector` stops it, which cancels the upgrade, then [calls `connector.start()` directly](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java#L787). That runs the new Connector code on the pre-upgrade flow. I'd put the guard in the node, either by refusing to start while an upgrade is pending or by having `start()` handle that case, so that every current and future caller is covered. ########## nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorParameterContextIT.java: ########## @@ -108,6 +109,20 @@ public void testParameterContextInheritanceWithSensitiveAndAssetParameters() thr assertEquals(SENSITIVE_SECRET_VALUE, sensitiveContent, "Sensitive output file should contain the secret value"); assertEquals(ASSET_FILE_CONTENT, assetContent, "Asset output file should contain the asset file contents"); + + // A same-bundle restart resumes the Connector with its sensitive and asset parameters still resolved + getNiFiInstance().stop(); Review Comment: [Claude Opus 5.5] This restart adds time to the suite but doesn't protect against the earlier bug where a normal restart restored the saved flow over the newly built one. The sensitive value here comes straight from the secret, so the saved flow and the newly built flow are the same, and the test would pass with or without that bug. I'd either drop this restart and cover that case with a unit test in `TestStandardConnectorRepository`, or use a value that the connector derives from the secret so that the restart actually checks something. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
