AW: Processor properties considered sensitive when migrating to Nifi 1.14.0

[christian.gumpert]

Hi Bryan,

thanks for your feedback. Case b) is exactly what happened to us. After a clean 
install containing only the official NARs and leaving our custom patches aside, 
Nifi started without a problem.

Sorry for the noise,

Christian Gumpert
Leiter BI & Analytics
Lead Data Scientist
Product Owner Team Lagrev

Eidgenössisches Finanzdepartement EFD
Eidgenössische Zollverwaltung EZV
Direktionsbereich Risikoanalyse und Analytik

Taubenstrasse 16, 3003 Bern
christian.gump...@ezv.admin.ch
www.ezv.admin.ch

-Ursprüngliche Nachricht-
Von: Bryan Bende  
Gesendet: Montag, 20. September 2021 15:06
An: users@nifi.apache.org
Cc: Trappenberg Jonas BIT ; Schneider Basil EZV 

Betreff: Re: Processor properties considered sensitive when migrating to Nifi 
1.14.0

Hello,

The reason for the error about the properties being sensitive is due to the 
ghost controller services... since a ghost controller service means we are 
missing the real definition of the component, NiFi doesn't know if the 
properties are actually sensitive or not, so it makes all properties on the 
ghost component sensitive to be cautious.

If you just upgraded from one version of nifi to the next, then you shouldn't 
have ghost components. Ghost components happen in one of two cases...

a) your flow.xml.gz references a component that doesn't exist in any of the NARs
b) your flow.xml.gz references a version of a component say X, and there are 
multiple versions available in the NARs, but none of them are X, for example 
versions Y and Z are available, but not X, so it doesn't know which one is 
correct.

Thanks,

Bryan

On Sun, Sep 19, 2021 at 3:05 PM  wrote:
>
> Dear Nifi experts,
>
>
>
> we are running a single-node Nifi installation with Nifi version 1.11.2 and 
> recently started to look into the upgrade to Nifi 1.14.0. While doing so we 
> observe one very odd behaviour when starting the new Nifi version with our 
> existing flow:
>
>
>
> During startup the Nifi initialization fails as some properties from 
> controller services or processors are set using parameters. According to the 
> error message (stack trace below) the property is considered being sensitive 
> (even though the documentation states it is not, e.g. “Database Connection 
> URL” in DBCPConnectionPool ) while the parameter is not sensitive. In our 
> humble opinion the error lies in the wrong assumption that the property is 
> sensitive.
>
>
>
> One thing we noticed is that there multiple instances of 
> GhostControllerServices being created because the old NAR versions are not 
> found anymore. We assume that this is normal behaviour.
>
>
>
> Things we have tried:
>
> · Starting Nifi with a clean/empty flow -> works
>
> · “Hack” the flow XML so that the parameter is defined as sensitive 
> -> kind of works as it falls over at the next parameter (we didn’t yet test 
> to make them all sensitive)
>
>
>
> Any help or suggestion is greatly appreciated,
>
> Christian
>
>
>
> Stacktrace:
>
>
>
> 2021-09-17 15:54:52,575 DEBUG [main] 
> o.a.n.c.v.StandardValidationTrigger Triggered to perform validation on 
> StandardControllerServiceNode[service=GhostControllerService[id=c86a30
> d7-ddbe-1257-57a4-32cd7854dd56, 
> type=org.apache.nifi.dbcp.DBCPConnectionPool], 
> name=TeradataConnection, active=false] asynchronously but flow is not 
> yet initialized so will ignore validation
>
> 2021-09-17 15:54:52,575 WARN [main] 
> org.eclipse.jetty.webapp.WebAppContext Failed startup of context 
> o.e.j.w.WebAppContext@1ab268bd{nifi-api,/nifi-api,file:///appl/nifi/ni
> fi-1.14.0/work/jetty/nifi-web-api-1.14.0.war/webapp/,UNAVAILABLE}{/app
> l/nifi/nifi-current/work/nar/extensions/nifi-server-nar-1.14.0.nar-unp
> acked/NAR-INF/bundled-dependencies/nifi-web-api-1.14.0.war}
>
> org.apache.nifi.controller.serialization.FlowSynchronizationException: 
> java.lang.IllegalArgumentException: The property 'Database Connection URL' is 
> a sensitive property, so it can only reference Parameters that are sensitive.
>
> at 
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowS
> ynchronizer.java:306)
>
> at 
> org.apache.nifi.controller.FlowController.synchronize(FlowController.j
> ava:1469)
>
> at 
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(Stand
> ardXMLFlowConfigurationDAO.java:89)
>
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardF
> lowService.java:810)
>
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowServic
> e.java:539)
>
> at 
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.
> contextInitialized(ApplicationStartupContextListener.java:67)
>
> at 
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized
> (ContextHandler.java:1068)
>
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized
> (ServletContextHandler.java:572)
>
>  

Processor properties considered sensitive when migrating to Nifi 1.14.0

[christian.gumpert]

Dear Nifi experts,

we are running a single-node Nifi installation with Nifi version 1.11.2 and 
recently started to look into the upgrade to Nifi 1.14.0. While doing so we 
observe one very odd behaviour when starting the new Nifi version with our 
existing flow:

During startup the Nifi initialization fails as some properties from controller 
services or processors are set using parameters. According to the error message 
(stack trace below) the property is considered being sensitive (even though the 
documentation states it is not, e.g. "Database Connection URL" in 
DBCPConnectionPool
 ) while the parameter is not sensitive. In our humble opinion the error lies 
in the wrong assumption that the property is sensitive.

One thing we noticed is that there multiple instances of 
GhostControllerServices being created because the old NAR versions are not 
found anymore. We assume that this is normal behaviour.

Things we have tried:

· Starting Nifi with a clean/empty flow -> works

· "Hack" the flow XML so that the parameter is defined as sensitive -> 
kind of works as it falls over at the next parameter (we didn't yet test to 
make them all sensitive)

Any help or suggestion is greatly appreciated,
Christian

Stacktrace:

2021-09-17 15:54:52,575 DEBUG [main] o.a.n.c.v.StandardValidationTrigger 
Triggered to perform validation on 
StandardControllerServiceNode[service=GhostControllerService[id=c86a30d7-ddbe-1257-57a4-32cd7854dd56,
 type=org.apache.nifi.dbcp.DBCPConnectionPool], name=TeradataConnection, 
active=false] asynchronously but flow is not yet initialized so will ignore 
validation
2021-09-17 15:54:52,575 WARN [main] org.eclipse.jetty.webapp.WebAppContext 
Failed startup of context 
o.e.j.w.WebAppContext@1ab268bd{nifi-api,/nifi-api,file:///appl/nifi/nifi-1.14.0/work/jetty/nifi-web-api-1.14.0.war/webapp/,UNAVAILABLE}{/appl/nifi/nifi-current/work/nar/extensions/nifi-server-nar-1.14.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.14.0.war}
org.apache.nifi.controller.serialization.FlowSynchronizationException: 
java.lang.IllegalArgumentException: The property 'Database Connection URL' is a 
sensitive property, so it can only reference Parameters that are sensitive.
at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1469)
at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at 
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
at 
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at 

AW: ExecuteSQL and Teradata

[christian.gumpert]

Hi Phil,

we are facing
Christian exactly the same issue (using Nifi 1.11.2) and I opened a ticket for 
this 
https://issues.apache.org/jira/browse/NIFI-8119#
I already have a patch for that locally and will submit a PR shortly.

Cheers,
Christian

Von: Toivo Adams 
Gesendet: Montag, 11. Januar 2021 21:24
An: users@nifi.apache.org
Betreff: Re: ExecuteSQL and Teradata

Hi,

And you are able to recognize you have received all data?
(My Teradata knowledge is limited, sorry.)
One solution is to create a customized version of ExecuteSQL.
And either close connection (return to pool) or send signal to Teradata.
This is not that hard.

You can also try to set QUERY_TIMEOUT for standard ExecuteSQL,
does Teradata JDBC support this?

BR
Toivo

Kontakt mailto:gravity.nif...@mailnull.com>> 
kirjutas kuupäeval R, 18. detsember 2020 kell 20:24:
Dear all

In the ExecuteSQL processor, I'm facing the following problem:

I want to execute a stored procedure in a Teradata database. This stored
procedure returns LOB data. Since I receive LOB data, I have LOB_SUPPORT on in
the JDBC driver (it's the default anyway). Since LOB data is not stored inline
in the database, Teradata expects a signal from the receiver that all data has
been received (in Teradata lingo that means KeepResp is on).

The problem now is, that NiFi does not send this signal. It keeps the connection
open. After NiFi has 16 connections to Teradata open, Teradata refuses to open
another connection and the following error is thrown:

[Teradata Database] : Response limit exceeded.

There's even a nice explanation about this error here [1].

I have set the maximum number of connections to 8, as is the default in the
controller. But that does not seem to prevent my issue. If I set a max timer for
the connections, it works, but of course I do not know how long I would need to
keep the connections open.

My question now is: How can I tell NiFi to close the connection to the Teradata
database, once it has received all data?

I appreciate all the help. Thanks,
Phil

[1] 
https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_5.html#CHDGCHBB


--
This message was sent from a MailNull anti-spam account.  You can get
your free account and take control over your email by visiting the
following URL.

   http://mailnull.com/