Re: NiFi 1.11.4 Upgrade

2020-07-16 Thread Bryan Bende 
Since NIFI-2215 has a fix version of 1.10.0, you could try going to that to
get the scripted processor improvements, but its possible NIFI-7460 also
affects the 1.10.0 release, I don't know without trying it out.

On Wed, Jul 15, 2020 at 5:09 PM Sri Harsha Chavali <
[email protected]> wrote:

> Hi Bryan,
>
> Thank you for sharing that. We did some testing of 1.11.4 on a server
> without versioned flows and liked all new features in 1.11.4 but
> unfortunately are not in a position to upgrade to 1.11.4 on our actual nifi
> servers. The features we are mainly looking for are the lifecycle methods
> on InvokeScriptedProcessor (
> https://issues.apache.org/jira/browse/NIFI-2215) and some other features.
> In order to get these what do you suggest is the best and stable version we
> can upgrade to from 1.9.2? We are really heavy on usage of custom
> processors. I know it's suggested to build our own full-blown processors
> but we really like the ease of use and code change using
> InvokeScriptedProcessor.
>
> Also, any suggestion to find all the versioned flows where we have NULL
> values in autoTerminatedRelationships and fixing them? We would like to
> see if they can be fixed on our side and still go with 1.11.4 deployment.
>
> Thank you,
> Harsha
>
> Sent from Outlook <http://aka.ms/weboutlook>
> --
> *From:* Bryan Bende 
> *Sent:* Wednesday, July 15, 2020 9:06 AM
> *To:* [email protected] 
> *Subject:* Re: NiFi 1.11.4 Upgrade
>
> Hello,
>
> Unfortunately this was a bug found after 1.11.4 and has been fixed in
> master, but is not in a release yet.
>
> https://issues.apache.org/jira/browse/NIFI-7460
>
> The community is hoping to release 1.12.0 soon though.
>
> Thanks,
>
> Bryan
>
>
> On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <
> [email protected]> wrote:
>
> Hi Users,
>
>
>
> I'm here with another question regarding the upgrade from NiFi-1.9.2 to
> NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting
> comfortable with the features, we have decided to upgrade our existing NiFi
> instance to 1.11.4. All of the process was smooth and I was able to get
> secure NiFi-1.11.4 up and running and then I get this message after logging
> into NiFi. Note that the authentication which uses LDAP has succeeded but
> then I get the screen which says:
>
>
>
> An unexpected error has occurred. Please check the logs for additional
> details.
>
>
>
> I copied users.xml, authorizations.xml as well as database_repository
> files from previous install to new installation. I have a separate location
> for other configurations like authorizers.xml,
> login-identity-providers.xml, state-management.xml, authorizers.xml,  
> flow.xml.gz
> and also the keystore and trustore files. I pointed the latest NiFi
> instance to the common config before starting it. Also, provenance, content
> and flowfile repositories are configured separately and I pointed new
> install to the existing locations.
>
>
> Given below are the error messages in logs after authentication is
> successful. My UserId is unable to access the root process group and
> basically nothing can be accessed in NiFi.
>
>
>
> nifi-user.log:
>
>
>
> 2020-07-14 13:41:18,996 INFO [NiFi Web Server-75]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication success for 
>
> 2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException. Returning Internal Server Error response.
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)
>
> at
> org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
>
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
> at
> j

Re: NiFi 1.11.4 Upgrade

2020-07-15 Thread Sri Harsha Chavali 
Hi Bryan,

Thank you for sharing that. We did some testing of 1.11.4 on a server without 
versioned flows and liked all new features in 1.11.4 but unfortunately are not 
in a position to upgrade to 1.11.4 on our actual nifi servers. The features we 
are mainly looking for are the lifecycle methods on InvokeScriptedProcessor 
(https://issues.apache.org/jira/browse/NIFI-2215) and some other features. In 
order to get these what do you suggest is the best and stable version we can 
upgrade to from 1.9.2? We are really heavy on usage of custom processors. I 
know it's suggested to build our own full-blown processors but we really like 
the ease of use and code change using InvokeScriptedProcessor.

Also, any suggestion to find all the versioned flows where we have NULL values 
in autoTerminatedRelationships and fixing them? We would like to see if they 
can be fixed on our side and still go with 1.11.4 deployment.

Thank you,
Harsha

Sent from Outlook<http://aka.ms/weboutlook>


From: Bryan Bende 
Sent: Wednesday, July 15, 2020 9:06 AM
To: [email protected] 
Subject: Re: NiFi 1.11.4 Upgrade

Hello,

Unfortunately this was a bug found after 1.11.4 and has been fixed in master, 
but is not in a release yet.

https://issues.apache.org/jira/browse/NIFI-7460

The community is hoping to release 1.12.0 soon though.

Thanks,

Bryan


On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali 
mailto:[email protected]>> wrote:

Hi Users,



I'm here with another question regarding the upgrade from NiFi-1.9.2 to 
NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting 
comfortable with the features, we have decided to upgrade our existing NiFi 
instance to 1.11.4. All of the process was smooth and I was able to get secure 
NiFi-1.11.4 up and running and then I get this message after logging into NiFi. 
Note that the authentication which uses LDAP has succeeded but then I get the 
screen which says:



An unexpected error has occurred. Please check the logs for additional details.



I copied users.xml, authorizations.xml as well as database_repository files 
from previous install to new installation. I have a separate location for other 
configurations like authorizers.xml, login-identity-providers.xml, 
state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and 
trustore files. I pointed the latest NiFi instance to the common config before 
starting it. Also, provenance, content and flowfile repositories are configured 
separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. 
My UserId is unable to access the root process group and basically nothing can 
be accessed in NiFi.



nifi-user.log:



2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] 
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for 

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] 
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

at 
org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

at 
org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..



nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] 
org.wali.MinimalLockingWriteAheadLog 
org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 
Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit 
Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] 
o.apache.nifi.controller.FlowController Failed to capture component stats for 
Stats History

java.lang.NullPointerException: null

at 
org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

at 
org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……

One thi

Re: NiFi 1.11.4 Upgrade

2020-07-15 Thread Bryan Bende 
Hello,

Unfortunately this was a bug found after 1.11.4 and has been fixed in
master, but is not in a release yet.

https://issues.apache.org/jira/browse/NIFI-7460

The community is hoping to release 1.12.0 soon though.

Thanks,

Bryan


On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <
[email protected]> wrote:

> Hi Users,
>
>
>
> I'm here with another question regarding the upgrade from NiFi-1.9.2 to
> NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting
> comfortable with the features, we have decided to upgrade our existing NiFi
> instance to 1.11.4. All of the process was smooth and I was able to get
> secure NiFi-1.11.4 up and running and then I get this message after logging
> into NiFi. Note that the authentication which uses LDAP has succeeded but
> then I get the screen which says:
>
>
>
> An unexpected error has occurred. Please check the logs for additional
> details.
>
>
>
> I copied users.xml, authorizations.xml as well as database_repository
> files from previous install to new installation. I have a separate location
> for other configurations like authorizers.xml,
> login-identity-providers.xml, state-management.xml, authorizers.xml,  
> flow.xml.gz
> and also the keystore and trustore files. I pointed the latest NiFi
> instance to the common config before starting it. Also, provenance, content
> and flowfile repositories are configured separately and I pointed new
> install to the existing locations.
>
>
> Given below are the error messages in logs after authentication is
> successful. My UserId is unable to access the root process group and
> basically nothing can be accessed in NiFi.
>
>
>
> nifi-user.log:
>
>
>
> 2020-07-14 13:41:18,996 INFO [NiFi Web Server-75]
> o.a.n.w.s.NiFiAuthenticationFilter Authentication success for 
>
> 2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException. Returning Internal Server Error response.
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)
>
> at
> org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
>
> at
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
>
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..
>
>
>
> nifi-app.log:
>
> 2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider
> Maintenance] org.wali.MinimalLockingWriteAheadLog
> org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records
> and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds,
> Clear Edit Logs time = 2 millis), max Transaction ID -1
>
> 2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.controller.FlowController Failed to capture component stats
> for Stats History
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)
>
> at
> org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……
>
> One thing to note is that the existing NiFi instance is installed a system
> service and we start and stop it using a sudo user. As this is the first
> time using the new instance, I didn't change the system service or init.d
> configuration and started nifi user. Also, the previous and current
> installs both have the below property in bootstrap.conf
>
> rus.as=nifi
>
> Could you please guide me what I'm doing wrong and what needs to be done? I'm
> hoping this is a simple authorization issue which can be fixed with a small
> change.
>
> Thank you,
> Harsha
>
> Sent from Outlook 
>

NiFi 1.11.4 Upgrade

2020-07-14 Thread Sri Harsha Chavali 
Hi Users,



I'm here with another question regarding the upgrade from NiFi-1.9.2 to 
NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting 
comfortable with the features, we have decided to upgrade our existing NiFi 
instance to 1.11.4. All of the process was smooth and I was able to get secure 
NiFi-1.11.4 up and running and then I get this message after logging into NiFi. 
Note that the authentication which uses LDAP has succeeded but then I get the 
screen which says:



An unexpected error has occurred. Please check the logs for additional details.



I copied users.xml, authorizations.xml as well as database_repository files 
from previous install to new installation. I have a separate location for other 
configurations like authorizers.xml, login-identity-providers.xml, 
state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and 
trustore files. I pointed the latest NiFi instance to the common config before 
starting it. Also, provenance, content and flowfile repositories are configured 
separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. 
My UserId is unable to access the root process group and basically nothing can 
be accessed in NiFi.



nifi-user.log:



2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] 
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for 

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] 
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

at 
org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

at 
org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..



nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] 
org.wali.MinimalLockingWriteAheadLog 
org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 
Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit 
Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] 
o.apache.nifi.controller.FlowController Failed to capture component stats for 
Stats History

java.lang.NullPointerException: null

at 
org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

at 
org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……

One thing to note is that the existing NiFi instance is installed a system 
service and we start and stop it using a sudo user. As this is the first time 
using the new instance, I didn't change the system service or init.d 
configuration and started nifi user. Also, the previous and current installs 
both have the below property in bootstrap.conf

rus.as=nifi

Could you please guide me what I'm doing wrong and what needs to be done? I'm 
hoping this is a simple authorization issue which can be fixed with a small 
change.

Thank you,
Harsha


Sent from Outlook