kevdoran opened a new pull request, #10909: URL: https://github.com/apache/nifi/pull/10909
<!-- Licensed to the Apache Software Foundation (ASF) under one or more --> <!-- contributor license agreements. See the NOTICE file distributed with --> <!-- this work for additional information regarding copyright ownership. --> <!-- The ASF licenses this file to You under the Apache License, Version 2.0 --> <!-- (the "License"); you may not use this file except in compliance with --> <!-- the License. You may obtain a copy of the License at --> <!-- http://www.apache.org/licenses/LICENSE-2.0 --> <!-- Unless required by applicable law or agreed to in writing, software --> <!-- distributed under the License is distributed on an "AS IS" BASIS, --> <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --> <!-- See the License for the specific language governing permissions and --> <!-- limitations under the License. --> # Summary [NIFI-15610](https://issues.apache.org/jira/browse/NIFI-15610) Extends the `ConnectorConfigurationProvider` extension point with asset management support, enabling external stores to manage connector assets (e.g., JDBC drivers, SSL certificates) alongside configuration. **Key changes:** - Added `storeAsset`, `loadAsset`, and `deleteAsset` methods to `ConnectorConfigurationProvider` for external asset CRUD - Extended `ConnectorConfigurationProvider`'s `ConnectorWorkingConfiguration` POJO with a list of `ConnectorAssetMetadata` (identifier, name, digest) so providers can describe externally-managed assets - Added asset lifecycle methods to `ConnectorRepository` (`storeAsset`, `getAsset`, `getAssets`, `deleteAssets`, `syncAssetsFromProvider`) and removed `getAssetRepository()` so that all asset operations are routed through the repository, which enforces provider synchronization - Implemented bidirectional ID translation in `StandardConnectorRepository` between NiFi asset UUIDs and external asset identifiers, since assets may originate from the external store with non-NiFi identifiers - Implemented provider-first asset upload with local rollback on failure, digest-based change detection (in-memory cache of provider-supplied digests), and on-demand binary sync during `applyUpdate` and `verifyConfigurationStep` to avoid unnecessary downloads of large binaries - Updated `StandardConnectorDAO` to sync assets from the provider before `verifyConfigurationStep`, ensuring asset binaries are available locally for verification - Updated `DtoFactory` and `WebApplicationConfiguration` to use `ConnectorRepository` directly instead of the now-encapsulated `ConnectorAssetRepository` ### Asset Upload Flow (provider-first) ```mermaid sequenceDiagram participant Client participant DAO as StandardConnectorDAO participant REPO as StandardConnectorRepository participant LOCAL as ConnectorAssetRepository<br/>(local files) participant CCP as ConnectorConfiguration-<br/>Provider Client->>DAO: createAsset(connectorId, assetId, name, content) DAO->>REPO: storeAsset(connectorId, assetId, name, content) REPO->>REPO: buffer content to byte[] REPO->>LOCAL: storeAsset(assetId, name, bytes) LOCAL-->>REPO: Asset (local file) opt provider configured REPO->>CCP: storeAsset(connectorId, name, bytes) CCP-->>REPO: externalId (e.g. stage path) REPO->>REPO: recordIdMapping(NiFi UUID ↔ externalId) end alt provider upload fails REPO->>LOCAL: deleteAsset(assetId) REPO-->>DAO: throw IOException end REPO-->>DAO: Asset ``` ### Asset Sync During Apply Update / Verify Step ```mermaid sequenceDiagram participant DAO as StandardConnectorDAO participant REPO as StandardConnectorRepository participant CCP as ConnectorConfiguration-<br/>Provider participant AM as AssetManager<br/>(local) DAO->>REPO: applyUpdate(connector, ctx)<br/>or syncAssetsFromProvider(connector) Note over REPO: syncFromProvider first:<br/>provider.load() returns config<br/>with asset metadata + digests REPO->>REPO: translateExternalToNifiIds()<br/>Match external IDs to NiFi UUIDs<br/>by property+name, or create placeholders REPO->>REPO: cache asset metadata with digests loop For each asset in cached metadata REPO->>REPO: lookup NiFi UUID for external ID alt local file missing OR digest changed OR no cached digest REPO->>CCP: loadAsset(connectorId, externalId) CCP-->>REPO: InputStream (binary content) REPO->>AM: storeAsset(nifiUuid, name, content) REPO->>REPO: update lastDownloadedProviderDigest else digest unchanged and file exists Note over REPO: skip download end end ``` ## Test Plan - [x] Unit tests for `StandardConnectorRepository` asset operations: store, get, delete, provider-first upload with rollback, external-to-NiFi ID translation, NiFi-to-external ID translation, digest-based sync skip, and asset cleanup - [x] Unit tests for `StandardConnectorDAO` verify-with-sync ordering and delete lifecycle - [x] All existing tests pass - [x] Checkstyle and PMD pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
