RE: Minifi 1.14.0 exception - sensitive props key

[Tomislav Novosel]

Hi Jeremy,

MiNiFi constructs nifi.properties file from the config.yml located in ./conf 
folder and this property needs to be set
manually in config.yml before the startup:

Security Properties:
  Sensitive Props:
key: ''

Thanks,
Tom

From: Jeremy Pemberton-Pigott 
Sent: 20 September 2021 16:58
To: users@nifi.apache.org
Subject: Re: Minifi 1.14.0 exception - sensitive props key

Hi Tom,

I recall that I used the NiFi 1.14.0 binary to set the key and then copied that 
over to the MiNiFi properties file to get it to work.

Linux only:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-key
./bin/nifi.sh set-sensitive-properties-key 

Jeremy

On Mon, Sep 20, 2021 at 9:14 PM Tomislav Novosel 
mailto:tomislav.novo...@clearpeaks.com>> wrote:
According to migration guidance for NiFi property 'Sensitive Properties Key' 
should be generated at startup, I believe this is the same behaviour in case of 
MiNiFi since they merged the codebase. Why is this happening?

BR,
Tom
From: Tomislav Novosel 
mailto:tomislav.novo...@clearpeaks.com>>
Sent: 20 September 2021 11:05
To: users@nifi.apache.org
Subject: Minifi 1.14.0 exception - sensitive props key

Hi to all,

I was using MiNiFi 0.5.0 running on ubuntu, installed as a service.
I switched now to MiNiFi 1.14.0 – I disabled minifi service, deleted
installation folder and unpacked MiNiFi 1.14.0 folder at the same place
where was 0.5.0 installed.

I started new MiNiFi 1.14.0 and the service don't want to start properly
with this exception:

java.lang.Exception: Unable to load flow due to: 
java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:166)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:163)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:64)
  at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265)
Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42)
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:125)
  ... 3 common frames omitted

I tried to set the key manually, but the property in nifi.properties gets 
overwritten
every time I am starting the service.

Could not find anything about this exception, can someone help, why is this 
happening?

Thanks, regards,
Tom

Re: Minifi 1.14.0 exception - sensitive props key

[Jeremy Pemberton-Pigott]

Hi Tom,

I recall that I used the NiFi 1.14.0 binary to set the key and then copied
that over to the MiNiFi properties file to get it to work.

Linux only:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-key
./bin/nifi.sh set-sensitive-properties-key 

Jeremy

On Mon, Sep 20, 2021 at 9:14 PM Tomislav Novosel <
tomislav.novo...@clearpeaks.com> wrote:

> According to migration guidance for NiFi property 'Sensitive Properties
> Key' should be generated at startup, I believe this is the same behaviour
> in case of MiNiFi since they merged the codebase. Why is this happening?
>
>
>
> BR,
>
> Tom
>
> *From:* Tomislav Novosel 
> *Sent:* 20 September 2021 11:05
> *To:* users@nifi.apache.org
> *Subject:* Minifi 1.14.0 exception - sensitive props key
>
>
>
> Hi to all,
>
>
>
> I was using MiNiFi 0.5.0 running on ubuntu, installed as a service.
>
> I switched now to MiNiFi 1.14.0 – I disabled minifi service, deleted
>
> installation folder and unpacked MiNiFi 1.14.0 folder at the same place
>
> where was 0.5.0 installed.
>
>
>
> I started new MiNiFi 1.14.0 and the service don't want to start properly
>
> with this exception:
>
>
>
> java.lang.Exception: Unable to load flow due to:
> java.lang.IllegalArgumentException: NiFi Sensitive Properties Key
> [nifi.sensitive.props.key] is required
>
>   at
> org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:166)
>
>   at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:163)
>
>   at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:64)
>
>   at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265)
>
> Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties
> Key [nifi.sensitive.props.key] is required
>
>   at
> org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42)
>
>   at
> org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:125)
>
>   ... 3 common frames omitted
>
>
>
> I tried to set the key manually, but the property in nifi.properties gets
> overwritten
>
> every time I am starting the service.
>
>
>
> Could not find anything about this exception, can someone help, why is
> this happening?
>
>
>
> Thanks, regards,
>
> Tom
>

AW: Processor properties considered sensitive when migrating to Nifi 1.14.0

[christian.gumpert]

Hi Bryan,

thanks for your feedback. Case b) is exactly what happened to us. After a clean 
install containing only the official NARs and leaving our custom patches aside, 
Nifi started without a problem.

Sorry for the noise,

Christian Gumpert
Leiter BI & Analytics
Lead Data Scientist
Product Owner Team Lagrev

Eidgenössisches Finanzdepartement EFD
Eidgenössische Zollverwaltung EZV
Direktionsbereich Risikoanalyse und Analytik

Taubenstrasse 16, 3003 Bern
christian.gump...@ezv.admin.ch
www.ezv.admin.ch

-Ursprüngliche Nachricht-
Von: Bryan Bende  
Gesendet: Montag, 20. September 2021 15:06
An: users@nifi.apache.org
Cc: Trappenberg Jonas BIT ; Schneider Basil EZV 

Betreff: Re: Processor properties considered sensitive when migrating to Nifi 
1.14.0

Hello,

The reason for the error about the properties being sensitive is due to the 
ghost controller services... since a ghost controller service means we are 
missing the real definition of the component, NiFi doesn't know if the 
properties are actually sensitive or not, so it makes all properties on the 
ghost component sensitive to be cautious.

If you just upgraded from one version of nifi to the next, then you shouldn't 
have ghost components. Ghost components happen in one of two cases...

a) your flow.xml.gz references a component that doesn't exist in any of the NARs
b) your flow.xml.gz references a version of a component say X, and there are 
multiple versions available in the NARs, but none of them are X, for example 
versions Y and Z are available, but not X, so it doesn't know which one is 
correct.

Thanks,

Bryan

On Sun, Sep 19, 2021 at 3:05 PM  wrote:
>
> Dear Nifi experts,
>
>
>
> we are running a single-node Nifi installation with Nifi version 1.11.2 and 
> recently started to look into the upgrade to Nifi 1.14.0. While doing so we 
> observe one very odd behaviour when starting the new Nifi version with our 
> existing flow:
>
>
>
> During startup the Nifi initialization fails as some properties from 
> controller services or processors are set using parameters. According to the 
> error message (stack trace below) the property is considered being sensitive 
> (even though the documentation states it is not, e.g. “Database Connection 
> URL” in DBCPConnectionPool ) while the parameter is not sensitive. In our 
> humble opinion the error lies in the wrong assumption that the property is 
> sensitive.
>
>
>
> One thing we noticed is that there multiple instances of 
> GhostControllerServices being created because the old NAR versions are not 
> found anymore. We assume that this is normal behaviour.
>
>
>
> Things we have tried:
>
> · Starting Nifi with a clean/empty flow -> works
>
> · “Hack” the flow XML so that the parameter is defined as sensitive 
> -> kind of works as it falls over at the next parameter (we didn’t yet test 
> to make them all sensitive)
>
>
>
> Any help or suggestion is greatly appreciated,
>
> Christian
>
>
>
> Stacktrace:
>
>
>
> 2021-09-17 15:54:52,575 DEBUG [main] 
> o.a.n.c.v.StandardValidationTrigger Triggered to perform validation on 
> StandardControllerServiceNode[service=GhostControllerService[id=c86a30
> d7-ddbe-1257-57a4-32cd7854dd56, 
> type=org.apache.nifi.dbcp.DBCPConnectionPool], 
> name=TeradataConnection, active=false] asynchronously but flow is not 
> yet initialized so will ignore validation
>
> 2021-09-17 15:54:52,575 WARN [main] 
> org.eclipse.jetty.webapp.WebAppContext Failed startup of context 
> o.e.j.w.WebAppContext@1ab268bd{nifi-api,/nifi-api,file:///appl/nifi/ni
> fi-1.14.0/work/jetty/nifi-web-api-1.14.0.war/webapp/,UNAVAILABLE}{/app
> l/nifi/nifi-current/work/nar/extensions/nifi-server-nar-1.14.0.nar-unp
> acked/NAR-INF/bundled-dependencies/nifi-web-api-1.14.0.war}
>
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalArgumentException: The property 'Database Connection URL' is 
> a sensitive property, so it can only reference Parameters that are sensitive.
>
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowS
> ynchronizer.java:306)
>
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.j
> ava:1469)
>
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Stand
> ardXMLFlowConfigurationDAO.java:89)
>
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardF
> lowService.java:810)
>
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowServic
> e.java:539)
>
> at 
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.
> contextInitialized(ApplicationStartupContextListener.java:67)
>
> at 
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized
> (ContextHandler.java:1068)
>
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized
> (ServletContextHandler.java:572)
>
>  

ApacheCon starts tomorrow!

[Rich Bowen]

ApacheCon @Home starts tomorrow! Details at 
https://www.apachecon.com/acah2021/index.html


(Note: You're receiving this because you are subscribed to one or more 
user lists for Apache Software Foundation projects.)


We've got three days of great content lined up for you, spanning 14 
project communities. And we're very excited about the keynotes, with 
presentations from David Nalley, Ashley Wolfe, Mark Cox, Alison Parker, 
and Michael Weinberg. And we'll be hearing from our Platinum sponsors in 
their keynotes as well! (Schedule is at 
https://www.apachecon.com/acah2021/tracks/)


You can still register today, at 
https://www.apachecon.com/acah2021/register.html


We especially want to thank our sponsors, who have made this event possible:

Strategic sponsor: Google
Platinum sponsors: Huawei, Tencent, Instaclustr, and Apple
Gold sponsors: AWS, Aiven, Gradle, Replicated, Red 
Hat, Baidu, Fiter, Cerner, Dremio, and Didi
Silver sponsors: Bamboo, SpereEx, Microsoft, Imply, Securonix, DataStax, 
and Crafter Software

Bronze sponsor: Technical Arts

Please join us on our Slack for discussion before, during, and after the 
event! http://s.apache.org/apachecon-slack


And follow us on Twitter - https://twitter.com/apachecon - for tips and 
announcements during the event.


See you tomorrow!


--
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

RE: Minifi 1.14.0 exception - sensitive props key

[Tomislav Novosel]

According to migration guidance for NiFi property 'Sensitive Properties Key' 
should be generated at startup, I believe this is the same behaviour in case of 
MiNiFi since they merged the codebase. Why is this happening?

BR,
Tom
From: Tomislav Novosel 
Sent: 20 September 2021 11:05
To: users@nifi.apache.org
Subject: Minifi 1.14.0 exception - sensitive props key

Hi to all,

I was using MiNiFi 0.5.0 running on ubuntu, installed as a service.
I switched now to MiNiFi 1.14.0 - I disabled minifi service, deleted
installation folder and unpacked MiNiFi 1.14.0 folder at the same place
where was 0.5.0 installed.

I started new MiNiFi 1.14.0 and the service don't want to start properly
with this exception:

java.lang.Exception: Unable to load flow due to: 
java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:166)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:163)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:64)
  at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265)
Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42)
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:125)
  ... 3 common frames omitted

I tried to set the key manually, but the property in nifi.properties gets 
overwritten
every time I am starting the service.

Could not find anything about this exception, can someone help, why is this 
happening?

Thanks, regards,
Tom

Re: Processor properties considered sensitive when migrating to Nifi 1.14.0

[Bryan Bende]

Hello,

The reason for the error about the properties being sensitive is due
to the ghost controller services... since a ghost controller service
means we are missing the real definition of the component, NiFi
doesn't know if the properties are actually sensitive or not, so it
makes all properties on the ghost component sensitive to be cautious.

If you just upgraded from one version of nifi to the next, then you
shouldn't have ghost components. Ghost components happen in one of two
cases...

a) your flow.xml.gz references a component that doesn't exist in any of the NARs
b) your flow.xml.gz references a version of a component say X, and
there are multiple versions available in the NARs, but none of them
are X, for example versions Y and Z are available, but not X, so it
doesn't know which one is correct.

Thanks,

Bryan

On Sun, Sep 19, 2021 at 3:05 PM  wrote:
>
> Dear Nifi experts,
>
>
>
> we are running a single-node Nifi installation with Nifi version 1.11.2 and 
> recently started to look into the upgrade to Nifi 1.14.0. While doing so we 
> observe one very odd behaviour when starting the new Nifi version with our 
> existing flow:
>
>
>
> During startup the Nifi initialization fails as some properties from 
> controller services or processors are set using parameters. According to the 
> error message (stack trace below) the property is considered being sensitive 
> (even though the documentation states it is not, e.g. “Database Connection 
> URL” in DBCPConnectionPool ) while the parameter is not sensitive. In our 
> humble opinion the error lies in the wrong assumption that the property is 
> sensitive.
>
>
>
> One thing we noticed is that there multiple instances of 
> GhostControllerServices being created because the old NAR versions are not 
> found anymore. We assume that this is normal behaviour.
>
>
>
> Things we have tried:
>
> · Starting Nifi with a clean/empty flow -> works
>
> · “Hack” the flow XML so that the parameter is defined as sensitive 
> -> kind of works as it falls over at the next parameter (we didn’t yet test 
> to make them all sensitive)
>
>
>
> Any help or suggestion is greatly appreciated,
>
> Christian
>
>
>
> Stacktrace:
>
>
>
> 2021-09-17 15:54:52,575 DEBUG [main] o.a.n.c.v.StandardValidationTrigger 
> Triggered to perform validation on 
> StandardControllerServiceNode[service=GhostControllerService[id=c86a30d7-ddbe-1257-57a4-32cd7854dd56,
>  type=org.apache.nifi.dbcp.DBCPConnectionPool], name=TeradataConnection, 
> active=false] asynchronously but flow is not yet initialized so will ignore 
> validation
>
> 2021-09-17 15:54:52,575 WARN [main] org.eclipse.jetty.webapp.WebAppContext 
> Failed startup of context 
> o.e.j.w.WebAppContext@1ab268bd{nifi-api,/nifi-api,file:///appl/nifi/nifi-1.14.0/work/jetty/nifi-web-api-1.14.0.war/webapp/,UNAVAILABLE}{/appl/nifi/nifi-current/work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.14.0.war}
>
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalArgumentException: The property 'Database Connection URL' is 
> a sensitive property, so it can only reference Parameters that are sensitive.
>
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
>
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1469)
>
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
>
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
>
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
>
> at 
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
>
> at 
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
>
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
>
> at 
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
>
> at 
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
>
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
>
> at 
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
>
> at 
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
>
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
>
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
>
> at 
> 

Minifi 1.14.0 exception - sensitive props key

[Tomislav Novosel]

Hi to all,

I was using MiNiFi 0.5.0 running on ubuntu, installed as a service.
I switched now to MiNiFi 1.14.0 - I disabled minifi service, deleted
installation folder and unpacked MiNiFi 1.14.0 folder at the same place
where was 0.5.0 installed.

I started new MiNiFi 1.14.0 and the service don't want to start properly
with this exception:

java.lang.Exception: Unable to load flow due to: 
java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:166)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:163)
  at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:64)
  at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265)
Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key 
[nifi.sensitive.props.key] is required
  at 
org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42)
  at 
org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:125)
  ... 3 common frames omitted

I tried to set the key manually, but the property in nifi.properties gets 
overwritten
every time I am starting the service.

Could not find anything about this exception, can someone help, why is this 
happening?

Thanks, regards,
Tom