[
https://issues.apache.org/jira/browse/NIFI-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713288#comment-17713288
]
Patrick A. Mol commented on NIFI-11464:
---------------------------------------
(It is a long story)
I used different names for clarity.
Using the template...
I added another avro reader service, named "same_name_avro_reader" to both the
process groups reusables and tenant_flows and enabled them.
Copied reusable_flow_Q to reusable_flow_R and changed the Record Reader
property to use the "same_name_avro_reader" service.
Started versioning on flow reusable_flow_R, then imported in tenant_flows.
It recognizes the controller service "same_name_avro_reader" on initial and
picks it up, only the Record Writer needs changing.
Connected ports to processor and funnels, so the whole package is in a valid
state with no invalid components.
Started versioning on flow tenant_flows.
Navigated to process group "production" and imported the flow.
This still gives me invalid controller services for both Record Reader and
Record Writer.
!image-2023-04-17-17-34-08-898.png!
The point is not that it picks up a controller service with the same name if
available after import, that works as intended.
The point is that when a nested versioned flow is part of another versioned
flow that gets imported,
the properties on the nested versioned flow appear to get the values recorded
in the "externalControllerServices" section of the nested flow's version.
(see reusable_flow_R-version-1.json.pretty below)
It does not look like the values it uses for the nested versioned flow are the
values the tenant_flow version was stored with (listed in the
controllerServices)
nor does it look like there is an attempt at resolving the controller server
reference by name.
Below snippets from nested versioned flow "reusable_flow_R" exported from
within "tenant_flows" in development.
(file reusable_flow_R_from_tenant_flows.json.pretty)
{code:java}
{
"flowContents": {
"identifier": "f5daaa02-42bf-35ce-8e5a-929c8546bd6f",
"instanceIdentifier": "01871025-5a73-1181-fc44-f06639beb421",
"name": "reusable_flow_R",
"comments": "used to perform R ...",
"position": {
"x": 472,
"y": 240
},
"processGroups": [],
"remoteProcessGroups": [],
"processors": [
{
"identifier": "27279fb9-5d37-3b5f-911d-5dc2ffa8fb6e",
"instanceIdentifier": "7afbc560-44a8-34a7-49de-521921cea5ee",
"name": "processor_with_controller_service(s)",
"comments": "",
"position": {
"x": 336,
"y": 240
},
"type": "org.apache.nifi.processors.standard.ConvertRecord",
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-nar",
"version": "1.21.0"
},
"properties": {
"record-writer": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
"record-reader": "aa305bc3-3743-3f7a-a780-cd806723fca0",
"include-zero-record-flowfiles": "true"
},
[...]
"funnels": [],
"controllerServices": [
{
"identifier": "aa305bc3-3743-3f7a-a780-cd806723fca0",
"instanceIdentifier": "01871024-5a73-1181-e8d5-3870646e2235",
"name": "same_name_avro_reader",
"comments": "",
"type": "org.apache.nifi.avro.AvroReader",
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-record-serialization-services-nar",
"version": "1.21.0"
},
[...]
},
{
"identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
"instanceIdentifier": "8f6cb319-0187-1000-b7fa-83340f7055f7",
"name": "tenant_flow_avro_writer",
"comments": "",
"type": "org.apache.nifi.avro.AvroRecordSetWriter",
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-record-serialization-services-nar",
"version": "1.21.0"
},
[...]{code}
and same but from tenant_flows imported in production (file
reusable_flow_R_in_production.json.pretty)
Note that the controllerServices section is empty.
{code:java}
{
"flowContents": {
"identifier": "f5daaa02-42bf-35ce-8e5a-929c8546bd6f",
"instanceIdentifier": "f41995a3-c208-3b26-db95-b89e1174f5b6",
"name": "reusable_flow_R",
"comments": "used to perform R ...",
"position": {
"x": 472,
"y": 240
},
"processGroups": [],
"remoteProcessGroups": [],
"processors": [
{
"identifier": "27279fb9-5d37-3b5f-911d-5dc2ffa8fb6e",
"instanceIdentifier": "daf9bfda-9694-3119-f874-6f911acd9176",
"name": "processor_with_controller_service(s)",
"comments": "",
"position": {
"x": 336,
"y": 240
},
"type": "org.apache.nifi.processors.standard.ConvertRecord",
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-nar",
"version": "1.21.0"
},
"properties": {
"record-writer": "b512b238-cdee-3642-b5cb-0c98d30dd133",
"record-reader": "29ea385c-b069-3981-afcf-f6032ad9bb3d",
"include-zero-record-flowfiles": "true"
},
[...]
"funnels": [],
"controllerServices": [],
"variables": {},
{code}
with exported flow version from registry as
reusable_flow_R-version-1.json.pretty
{code:java}
$ head -15 reusable_flow_R-version-1.json.pretty
{
"externalControllerServices": {
"b512b238-cdee-3642-b5cb-0c98d30dd133": {
"identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
"name": "reusables_avro_writer"
},
"29ea385c-b069-3981-afcf-f6032ad9bb3d": {
"identifier": "29ea385c-b069-3981-afcf-f6032ad9bb3d",
"name": "same_name_avro_reader"
}
},
"flowContents": {
"comments": "used to perform R ...",
"componentType": "PROCESS_GROUP",
"connections": [
{code}
If I look for the controller service identifier as recorded in the version in
the valid tenant_flows process group (downloaded to
reusable_flow_R_from_tenant_flows.json.pretty)
it is not there.
{code:java}
$ fgrep '29ea385c-b069-3981-afcf-f6032ad9bb3d'
reusable_flow_R_from_tenant_flows.json.pretty
${code}
So it definitely needs to go to the Registry to fetch the nested versioned flow.
And it does, because when I delete reusable_flow_R from the registry I get this
!image-2023-04-17-18-06-16-102.png!
I would argue that you should always be able to retrieve a versioned flow from
the registry,
even when a nested versioned flow is no longer available, for whatever reason,
because you could use a unknowingly lose a lot of work.
> importing a versioned flow with a nested versioned flow shows nested
> versioned flow with invalid controller services.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-11464
> URL: https://issues.apache.org/jira/browse/NIFI-11464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 1.21.0
> Environment: nifi 1.21.0 and nifi registry 1.21.0 (on ubuntu 20.04)
> Reporter: Patrick A. Mol
> Priority: Major
> Attachments: exported_flow_versions_pretty.zip,
> image-2023-04-17-17-34-08-898.png, image-2023-04-17-18-06-16-102.png,
> nested_versioned_flow_issue.xml, screenprints_reproduction_steps.zip
>
>
> When a flow (reusable_flow_Q) has controller services inherited from the
> hierarchy (process group reusables) and a version of the flow is stored, the
> flow version contains the references to these external controller services
> (as seen in an exported flow version [see below]).
> When this versioned flow is imported in another flow (tenant_flows) the
> controller services need to be reset to the controller services in the new
> hierarchy.
> When we have a working flow with the nested versioned flow ready in
> development we can check this flow into version control.
> When we then deploy the flow in production, the nested versioned flow shows
> up with invalid components. It shows the external controller service
> identifiers as stored in the flow version.
> When we then go back to development version of tenant_flows and make a minor
> change to the nested versioned flow reusable_flow_Q and commit this change to
> version control.
> Due to this version change, we need to also commit the changes for the
> tenant_flows process group.
> When we now go back to production, and import this new version of
> tenant_flows, the nested versioned flow reusable_flow_Q does not have invalid
> controller services.
> If you have several flows under development using the same reusable
> components,
> you will likely end up with invalid components after import.
> Depending on the amount of versioned flows used, it could be a lot of work.
> It could also lead to issues when using the ExecuteStateless processor.
> Please see attached template nested_version_flow_issue.xml for a starting
> point to reproduce the issue. It contains the steps.
> Screenprints are attached in a zip file show the process and diagnosis.
> Controller services identifiers in version 2.
> {code:java}
> $ fgrep -C 4 reusables_avro reusable_flow_Q.json.pretty
> "controllerServices": [
> {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "instanceIdentifier": "8f647d06-0187-1000-4be9-14a61f55d904",
> "name": "reusables_avro_reader",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroReader",
> "bundle": {
> "group": "org.apache.nifi",
> --
> },
> {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "instanceIdentifier": "8f64f2c6-0187-1000-7557-ca63c88054dd",
> "name": "reusables_avro_writer",
> "comments": "",
> "type": "org.apache.nifi.avro.AvroRecordSetWriter",
> "bundle": {
> "group": "org.apache.nifi",
> $ head -15 reusable_flow_Q-version-2.json.pretty
> {
> "externalControllerServices": {
> "dc884171-4d75-3854-8604-afab91bd0e60": {
> "identifier": "dc884171-4d75-3854-8604-afab91bd0e60",
> "name": "reusables_avro_reader"
> },
> "b512b238-cdee-3642-b5cb-0c98d30dd133": {
> "identifier": "b512b238-cdee-3642-b5cb-0c98d30dd133",
> "name": "reusables_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
> Controller services identifiers with version 3 committed in process group
> "tenant_flows".
> {code:java}
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ fgrep -C
> 4 tenant_flow_avro tenant_flows-version-1.json.pretty
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "instanceIdentifier": "8f6cb319-0187-1000-b7fa-83340f7055f7",
> "name": "tenant_flow_avro_writer",
> "properties": {
> "compression-format": "NONE",
> "Schema Write Strategy": "avro-embedded",
> "schema-name": "${schema.name}",
> --
> ],
> "groupIdentifier": "a984831b-8587-3e17-bbbc-ef4b85c3898d",
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "instanceIdentifier": "8f6c8a94-0187-1000-af54-2fee12838618",
> "name": "tenant_flow_avro_reader",
> "properties": {
> "schema-name": "${schema.name}",
> "cache-size": "1000",
> "schema-access-strategy": "embedded-avro-schema",
> pmo@hpmo:~/Documents.local/nested_versioned_flows_controller_issue$ head -15
> reusable_flow_Q-version-3.json.pretty
> {
> "externalControllerServices": {
> "8ff96d88-3dc8-30ed-aeb8-757c26a7b807": {
> "identifier": "8ff96d88-3dc8-30ed-aeb8-757c26a7b807",
> "name": "tenant_flow_avro_reader"
> },
> "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22": {
> "identifier": "5d9df37d-2a52-3f6e-8cd3-3d3ea9550d22",
> "name": "tenant_flow_avro_writer"
> }
> },
> "flowContents": {
> "comments": "used to perform Q ...",
> "componentType": "PROCESS_GROUP",
> "connections": [
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
