[
https://issues.apache.org/jira/browse/NIFI-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713269#comment-17713269
]
Bryan Bende commented on NIFI-11464:
------------------------------------
Thanks for the detailed write up and detailed template. From playing around
with the template, I think it is working as it is currently designed, although
there could possibly some debate whether an improvement could be made.
The idea of resolving external services originally came from a scenario like
the following:
* Dev NiFi has a service at root PG named "Schema Registry"
* Some processor in a versioned controlled flow uses "Schema Registry"
* Prod NiFi also has a service at root PG named "Schema Registry" of same
service type
* Someone imports the dev flow to prod, it looks for a service in the parent
group with the same name and type, if there is only one it auto-selects it
The key point here is the criteria is "same name AND type".
In your example, the first step saves a flow to registry where the services are
named "reusable avro reader" and "reusable avro writer". In step 3, it then
imports into the tenants PG where there are existing services of matching
types, however they are named "tenants reader" and "tenants writer", so the way
it is implemented, these won't be selected since the names are different. Then
when you select the correct tenant services and save the resuable flow to
registry again, now the external services are saved with the tenant names, so
then next time you import the overall tenants flow now everything lines up.
I can see an argument that the name should only be used when there is more than
one choice to choose from, but currently it looks for a service where name and
type match together.
> importing a versioned flow with a nested versioned flow shows nested
> versioned flow with invalid controller services.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-11464
> URL: https://issues.apache.org/jira/browse/NIFI-11464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 1.21.0
> Environment: nifi 1.21.0 and nifi registry 1.21.0 (on ubuntu 20.04)
> Reporter: Patrick A. Mol
> Priority: Major
> Attachments: exported_flow_versions_pretty.zip,
> nested_versioned_flow_issue.xml, screenprints_reproduction_steps.zip
>
>
> When a flow (reusable_flow_Q) has controller services inherited from the
> hierarchy (process group reusables) and a version of the flow is stored, the
> flow version contains the references to these external controller services
> (as seen in an exported flow version [see below]).
> When this versioned flow is imported in another flow (tenant_flows) the
> controller services need to be reset to the controller services in the new
> hierarchy.
> When we have a working flow with the nested versioned flow ready in
> development we can check this flow into version control.
> When we then deploy the flow in production, the nested versioned flow shows
> up with invalid components. It shows the external controller service
> identifiers as stored in the flow version.
> When we then go back to development version of tenant_flows and make a minor
> change to the nested versioned flow reusable_flow_Q and commit this change to
> version control.
> Due to this version change, we need to also commit the changes for the
> tenant_flows process group.
> When we now go back to production, and import this new version of
> tenant_flows, the nested versioned flow reusable_flow_Q does not have invalid
> controller services.
> If you have several flows under development using the same reusable
> components,
> you will likely end up with invalid components after import.
> Depending on the amount of versioned flows used, it could be a lot of work.
> It could also lead to issues when using the ExecuteStateless processor.
> Please see attached template nested_version_flow_issue.xml for a starting
> point to reproduce the issue. It contains the steps.
> Screenprints are attached in a zip file show the process and diagnosis.
> Controller services identifiers in version 2.
> {code:java}
> $ fgrep -C 4 reusables_avro reusable_flow_Q.json.pretty
> "controllerServices": [
> {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "instanceIdentifier": "8f647d06-0187-1000-4be9-14a61f55d904",
> "name": "reusables_avro_reader",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroReader",
> "bundle": {
> "group": "org.apache.nifi",
> --
> },
> {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "instanceIdentifier": "8f64f2c6-0187-1000-7557-ca63c88054dd",
> "name": "reusables_avro_writer",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroRecordSetWriter",
> "bundle": {
> "group": "org.apache.nifi",
> $ head -15 reusable_flow_Q-version-2.json.pretty
> {
> "externalControllerServices": {
> "dc884171-4d75-3854-8604-afab91bd0e60": {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "name": "reusables_avro_reader"
> },
> "b512b238-cdee-3642-b5cb-0c98d30dd133": {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "name": "reusables_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
> Controller services identifiers with version 3 committed in process group
> "tenant_flows".
> {code:java}
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ fgrep -C
> 4 tenant_flow_avro tenant_flows-version-1.json.pretty
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "instanceIdentifier": "8f6cb319-0187-1000-b7fa-83340f7055f7",
> "name": "tenant_flow_avro_writer",
> "properties": {
> "compression-format": "NONE",
> "Schema Write Strategy": "avro-embedded",
> "schema-name": "${schema.name}",
> --
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "instanceIdentifier": "8f6c8a94-0187-1000-af54-2fee12838618",
> "name": "tenant_flow_avro_reader",
> "properties": {
> "schema-name": "${schema.name}",
> "cache-size": "1000",
> "schema-access-strategy": "embedded-avro-schema",
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ head -15
> reusable_flow_Q-version-3.json.pretty
> {
> "externalControllerServices": {
> "8ff96d88-3dc8-30ed-aeb8-757c26a7b807": {
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "name": "tenant_flow_avro_reader"
> },
> "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22": {
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "name": "tenant_flow_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
