[
https://issues.apache.org/jira/browse/NIFI-12918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831362#comment-17831362
]
Stephanie Ambrose edited comment on NIFI-12918 at 3/27/24 2:05 PM:
-------------------------------------------------------------------
Okay, so after some back and forth here is my final conclusion to this bug:
I've submitted two PRs
[https://github.com/apache/nifi/pull/8572/checks] <-- for support/1.x branch
[https://github.com/apache/nifi/pull/8536/checks] <-- for main branch
Here is what's happening. If you run a versioned stateless flow that has
sub-versioned progress group, there is a NullPointerException thrown on the
build method of StandardVersionControlInformation where there is a non-null
requirement on "registryId".
Code has changed in main versus support/1.x, but the bug exists on both. What
I've found is the JerseyClient calls that are made to map the response from the
registry-api back to the higher level VersionedProcessGroup ->
VersionedFlowCoordinates both return JSON where the VersionedFlowCoordinates >
registryId is always null. Now in the main branch, stateless uses some of these
new "synchronizer" classes, so the best place to insert a solution is to add an
else on the null check where the code runs the "determineRegistryId" method and
set the value to "1" for versioned flows that do not have a registryId
associated to them. Doing this passes all of the integration tests and code
checks and fixes the bug. Since this class does not exist in the 1.x support
branch, my best solution is to just comment out the null check (this breaks
integration tests in main, but not in 1.x).
I continued to look further at this idea of a "registryId" within the registry
api code. The GET method on buckets/flow/version returns the same class that
the stateless code uses to map into on the JerseyClient call. I then looked at
the api code to POST new versioned flows, and it also expect the json as a
parameter to match the same class. Well, since there is no "non-null"
requirement on the registryId, versioned flows are stored in the database (or
whichever storage adapter used) with this property as null. I did not check to
see if the latest NiFi Registry UI is now setting this property, but making it
non-null would break any older version of registry and not be backwards
compatible. Therefore, the only solution at this point to fix stateless NiFi in
its current state is to just set registryId to "1" if it is null. I believe
this code is still probably prototype being worked and evolving, so this will
be a temporary fix until those mandates are in place on a concept of
"registryId".
[~markap14]
was (Author: JIRAUSER299049):
Okay, so after some back and forth here is my final conclusion to this bug:
I've submitted two PRs
[https://github.com/apache/nifi/pull/8572/checks] <-- for support/1.x branch
[https://github.com/apache/nifi/pull/8536/checks] <-- for main branch
Here is what's happening. If you run a versioned stateless flow that has
sub-versioned progress group, there is a NullPointerException thrown on the
build method of StandardVersionControlInformation where there is a non-null
requirement on "registryId".
Code has changed in main versus support/1.x, but the bug exists on both. What
I've found is the JerseyClient calls that are made to map the response from the
registry-api back to the higher level VersionedProcessGroup ->
VersionedFlowCoordinates both return JSON where the VersionedFlowCoordinates >
registryId is always null. Now in the main branch, stateless uses some of these
new "synchronizer" classes, so the best place to insert a solution is to add an
else on the null check where the code runs the "determineRegistryId" method and
set the value to "1" for versioned flows that do not have a registryId
associated to them. Doing this passes all of the integration tests and code
checks and fixes the bug. Since this class does not exist in the 1.x support
branch, my best solution is to just comment out the null check (this breaks
integration tests in main, but not in 1.x).
I continued to look further at this idea of a "registryId" within the registry
api code. The GET method on buckets/flow/version returns the same class that
the stateless code uses to map into on the JerseyClient call. I then looked at
the api code to POST new versioned flows, and it also expect the json as a
parameter to match the same class. Well, since there is no "non-null"
requirement on the registryId, versioned flows are stored in the database (or
whichever storage adapter used) with this property as null. I did not check to
see if the latest NiFi Registry UI is now setting this property, but making it
non-null would break any older version of registry and not be backwards
compatible. Therefore, the only solution at this point to fix stateless NiFi in
its current state is to just set registryId to "1" if it is null. I believe
this code is still probably prototype being worked and evolving, so this will
be a temporary fix until those mandates are in place on a concept of
"registryId".
> Stateless NiFi NullPointerException on versioned PGs
> ----------------------------------------------------
>
> Key: NIFI-12918
> URL: https://issues.apache.org/jira/browse/NIFI-12918
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Stephanie Ambrose
> Assignee: Stephanie Ambrose
> Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> If you create a process group and commit that to version control, and then
> add a sub-process group under that one and commit that to its own version
> control, it will throw a NullPointerException:
> *NOTE: This error only occurs for stateless nifi processing.*
> {code:java}
> 2024-03-15T12:25:38.565156967Z 2024-03-15 12:25:38,564 INFO [main]
> o.a.nifi.groups.StandardProcessGroup
> StandardProcessGroup[identifier=9688e64b-3f4a-3668-81b9-e25d9ea61454,name=Sub
> Group] added to StandardProcessGroup[identifier=stateless-flow,name=Test Flow]
> 2024-03-15T12:25:38.570693223Z Exception in thread "main"
> java.lang.NullPointerException: Registry ID must be specified
> at java.base/java.util.Objects.requireNonNull(Unknown Source)
> 2024-03-15T12:25:38.570722016Z at
> org.apache.nifi.registry.flow.StandardVersionControlInformation$Builder.build(StandardVersionControlInformation.java:134)
> 2024-03-15T12:25:38.570725111Z at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:354)
> 2024-03-15T12:25:38.570729455Z at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.addProcessGroup(StandardVersionedComponentSynchronizer.java:1185)
> at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronizeChildGroups(StandardVersionedComponentSynchronizer.java:528)
> 2024-03-15T12:25:38.573083029Z at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:426)
> 2024-03-15T12:25:38.573086447Z at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.lambda$synchronize$0(StandardVersionedComponentSynchronizer.java:265)
> 2024-03-15T12:25:38.573092747Z at
> org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:551)
> 2024-03-15T12:25:38.573095677Z at
> org.apache.nifi.flow.synchronization.StandardVersionedComponentSynchronizer.synchronize(StandardVersionedComponentSynchronizer.java:260)
> 2024-03-15T12:25:38.573098999Z at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3977)
> at
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3957)
> at
> org.apache.nifi.stateless.engine.StandardStatelessEngine.createFlow(StandardStatelessEngine.java:180)
> 2024-03-15T12:25:38.573108177Z at
> org.apache.nifi.stateless.flow.StandardStatelessDataflowFactory.createDataflow(StandardStatelessDataflowFactory.java:243)
> at
> org.apache.nifi.stateless.bootstrap.StatelessBootstrap.createDataflow(StatelessBootstrap.java:73)
> {code}
> When stepping through the debugger, it appears the "registryId" property is
> null. The JerseyClient call is supposed to map the response from registry to
> the proper class, but registryId is never set:
> {code:java}
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.nifi.flow.VersionedProcessGroup.setVersionedFlowCoordinates(VersionedProcessGroup.java:150)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Unknown
> Source:-1)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source:-1)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source:-1)
> at java.lang.reflect.Method.invoke(Unknown Source:-1)
> at
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:141)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:359)
> at
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
> at
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
> at
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
> at
> com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
> at
> com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:2079)
> at
> com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1229)
> at
> org.glassfish.jersey.jackson.internal.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:829)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:233)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:212)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
> at
> org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:298)
> at
> org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:742)
> at
> org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675)
> at
> org.glassfish.jersey.client.JerseyInvocation$$Lambda$294.273041802.call(Unknown
> Source:-1)
> at
> org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
> at
> org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
> at
> org.glassfish.jersey.client.JerseyInvocation$$Lambda$295.608108604.call(Unknown
> Source:-1)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
> at
> org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
> at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422)
> at
> org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318)
> at
> org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.lambda$get$1(JerseyFlowSnapshotClient.java:110)
> at
> org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient$$Lambda$191.1164622694.execute(Unknown
> Source:-1)
> at
> org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:103)
> at
> org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.get(JerseyFlowSnapshotClient.java:103)
> at
> org.apache.nifi.stateless.core.RegistryUtil.getFlowContents(RegistryUtil.java:113)
> at
> org.apache.nifi.stateless.core.RegistryUtil.getFlowByID(RegistryUtil.java:61)
> at
> org.apache.nifi.stateless.config.PropertiesFileFlowDefinitionParser.fetchFlowFromRegistry(PropertiesFileFlowDefinitionParser.java:603)
> at
> org.apache.nifi.stateless.config.PropertiesFileFlowDefinitionParser.fetchVersionedFlowSnapshot(PropertiesFileFlowDefinitionParser.java:541)
> at
> org.apache.nifi.stateless.config.PropertiesFileFlowDefinitionParser.parseFlowDefinition(PropertiesFileFlowDefinitionParser.java:119)
> at
> org.apache.nifi.stateless.config.PropertiesFileFlowDefinitionParser.parseFlowDefinition(PropertiesFileFlowDefinitionParser.java:106)
> at
> org.apache.nifi.stateless.bootstrap.StatelessBootstrap.parseDataflowDefinition(StatelessBootstrap.java:80)
> at
> org.apache.nifi.stateless.bootstrap.RunStatelessFlow.createDataflow(RunStatelessFlow.java:92)
> at
> org.apache.nifi.stateless.bootstrap.RunStatelessFlow.main(RunStatelessFlow.java:56)
> {code}
>
> Looking further, it appears the "storageLocation" property is actually the
> dependency that is used for the sub-getFlow calls. I submitted the PR to
> change the non-null requirement to storageLocation for this reason. The
> sub-call in 1.x (1.25 tested) also uses the wrong "getRegistryUrl" method,
> which was changed to "getStorageLocation" on the main branch. Making these
> two changes fixes the versioned PG bug in stateless NiFi. Ultimately, it
> looks like
> VersionedFlowCoordinates are not mapped from the JerseyClient call correctly.
> This is not done directly from the JSON returned from the registry-api call.
> (I'm not sure where that's done)...but with other places showing "registryId"
> as "placeholder", I'm assuming this is something still in "to do" status.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
