[
https://issues.apache.org/jira/browse/NIFI-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess resolved NIFI-8684.
--------------------------------
Fix Version/s: 1.15.0
Assignee: Matt Burgess
Resolution: Fixed
Closing as fixed by NIFI-7012
> sensitive property not working for InvokeScriptedProcessor
> ----------------------------------------------------------
>
> Key: NIFI-8684
> URL: https://issues.apache.org/jira/browse/NIFI-8684
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Reporter: Jul Tomten
> Assignee: Matt Burgess
> Priority: Major
> Labels: InvokeScriptedProcessor, property, sensitive
> Fix For: 1.15.0
>
>
> I use InvokeScriptedProcessor
> I'm trying to read a sensitive property from the process context
>
> before restarting NiFi it was working fine
> after restarting NiFi - NiFi fails to startup with the error below
> see https://issues.apache.org/jira/browse/NIFI-7012
>
>
> 2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalArgumentException: The property 'Password3' cannot reference
> Parameter 'password3' because Sensitive Parameters may only be referenced by
> Sensitive Properties.
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> at
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:423)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.Server.doStart(Server.java:387)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
> at org.apache.nifi.NiFi.<init>(NiFi.java:159)
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> at org.apache.nifi.NiFi.main(NiFi.java:303)
> Caused by: java.lang.IllegalArgumentException: The property 'Password3'
> cannot reference Parameter 'password3' because Sensitive Parameters may only
> be referenced by Sensitive Properties.
> at
> org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
> at
> org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
> at
> org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
> ... 43 common frames omitted
>
> in flow.xml.gz
>
>
> {code:java}
> <parameterContext>
> <id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
> <name>otto_test</name>
> <description/>
> <parameter>
> <name>password</name>
> <description/>
> <sensitive>true</sensitive>
> <value>enc
> {61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
> </value>
> </parameter>
> <parameter>
> <name>password3</name>
> <description/>
> <sensitive>true</sensitive>
> <value>enc
> {6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
> </value>
> </parameter>
> </parameterContext>
> {code}
>
>
>
>
>
>
> {code:java}
> JIRA messed up and inserted lots of brackets {{ and }} all over the place, I
> tried to remove them but may have not got them all or to many removed{code}
> {code:java}
> <processor>
> <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
> <name>InvokeScriptedProcessor2</name>
> <position x="-1144.0" y="3640.0"/>
> <styles/>
> <comment/>
>
> <class&gt;org.apache.nifi.processors.script.InvokeScriptedProcessor</class&gt;
> <bundle>
> <group>org.apache.nifi</group>
> <artifact>nifi-scripting-nar</artifact>
> <version>1.13.2</version>
> </bundle>
> <maxConcurrentTasks>1</maxConcurrentTasks>
> <schedulingPeriod>0 sec</schedulingPeriod>
> <penalizationPeriod>30 sec</penalizationPeriod>
> <yieldPeriod>1 sec</yieldPeriod>
> <bulletinLevel>WARN</bulletinLevel>
> <lossTolerant>false</lossTolerant>
> <scheduledState>RUNNING</scheduledState>
> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
> <executionNode>ALL</executionNode>
> <runDurationNanos>0</runDurationNanos>
> <property>
> <name>Script Engine</name>
> <value>ECMAScript</value>
> </property>
> <property>
> <name>Script File</name>
> </property>
> <property>
> <name>Script Body</name>
>
> <value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
>
> //http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
>
> //http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
>
> //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
> ///////////////////////////////////////////////////////////
> // "imports" go here
> ////////////////////////////////////////////////////////////
> var StreamCallback =
> Java.type("org.apache.nifi.processor.io.StreamCallback");
> var IOUtils = Java.type("org.apache.commons.io.IOUtils");
> var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
> var Processor = Java.type("org.apache.nifi.processor.Processor");
> var Relationship = Java.type("org.apache.nifi.processor.Relationship");
> var StandardValidators =
> Java.type("org.apache.nifi.processor.util.StandardValidators");
> var Validator = Java.type("org.apache.nifi.components.Validator");
>
> var HashSet = Java.type("java.util.HashSet");
> var LinkedList = Java.type("java.util.LinkedList");
> var PropertyDescriptorBuilder =
> Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
> var log = null;
> var REL_SUCCESS = new
> Relationship.Builder().name("success").description('FlowFiles that were
> successfully processed are routed here').build();
> var REL_FAILURE = new
> Relationship.Builder().name("failure").description('FlowFiles that were not
> successfully processed are routed here').build();
> var strPassword2 = "Password2";
> var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
> .description("Password used to connect")
> .required(true)
> //.sensitive(false)
> //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(true)
> .addValidator(Validator.VALID)
> .build();
> var strPassword3 = "Password3";
> var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
> .description("Password used to connect")
> .required(true)
> .sensitive(true)}}
> // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(false)
> .addValidator(Validator.VALID)
> .build();
> function executeScript(session, context, log, REL_SUCCESS, REL_FAILURE) {
> ////////////////////////////////////////////////////////////
> // your code goes here
> ////////////////////////////////////////////////////////////
> var pwd2 = context.getProperty(strPassword2).getValue();
> var pwd3 = context.getProperty(strPassword3).getValue();
> var pwd31 =
> context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
> log.error('pwd2:'+ pwd2);
> log.error('pwd3:'+ pwd3);
> var flowFile = session.get();
> if(flowFile != null) {}}if(flowFile != null) {
> // Create a new StreamCallback, passing in a function to define the
> interface method
> flowFile = session.write(flowFile,
> new StreamCallback(function(inputStream, outputStream){
> var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
> outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8));
>
> outputStream.write("\npwd2:".getBytes());
> outputStream.write(pwd2.getBytes());
> // outputStream.write("\npwd3:".getBytes());
> // outputStream.write(pwd3.getBytes());
> // outputStream.write("\n31:".getBytes());
> // outputStream.write(pwd31.getBytes());
> outputStream.write("\n".getBytes()); }
> ));
>
> try{
> session.transfer(flowFile, REL_SUCCESS)
> }catch(e){
> log.error('Something went wrong', e)
> session.transfer(flowFile, REL_FAILURE)
> }
> }
> function initialize(context){ log = context.logger; }
> function getRelationships()
> {
> var r = new HashSet();
> r.add(REL_FAILURE);
> r.add(REL_SUCCESS);
> return r;
> }
> function validate(context){ return null; }
> function getPropertyDescriptor(name) {
> if(name.equals(strPassword2)){
> return prop_password2;
> } else if(name.equals(strPassword3)){
> return prop_password3;
> } else{{
> return null;
> }
> }
> function onPropertyModified(descriptor, oldValue, newValue) { return null; }
> function getPropertyDescriptors(){
> var r = new LinkedList();
> r.add(prop_password2);
> r.add(prop_password3);
> return r;
> }
> function getIdentifier(){ return null; }
> function onTrigger(context, sessionFactory) {
> var session = sessionFactory.createSession();
> try{
> executeScript(session, context, log, REL_SUCCESS, REL_FAILURE);
> session.commit();
> }catch (t) {
> log.error("{} failed to process due to {}; rolling back session",
> Java.to([this, t], "java.lang.Object[]"));
> session.rollback(true);
> throw t;
> }
> }
> processor = this;</value>
> </property>
> <property>
> <name>Module Directory</name>
> </property>
> <property>
> <name>Password2</name>
> <value>xyz</value>
> </property>
> <property>
> <name>Password3</name>
> <value>enc
> {da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
> </value>
> </property>
> </processor>
>
> {code}
>
> to recover I had to uncomment the code and remove the property: Password3
> using a text editor on the flow.xml
>
> Many thanks to mburgess the cookbook author.
> I'm new to NiFi so sorry if I've simply made a programming error.
> It seems like I'm not the only one with the similar problem.
> I try to createa flow that login to provenance repository in a NiFi with LDAP
> using the InvokeHttp in a later step. To login I need to fetch a JWT-token
> and then call the provenanc repo REST apis using the token. When fetching the
> token I want to hide tha password. Maybe there is a better way. I also would
> like to store the token in a secure way and not store it as attribute. I use
> the DistributedMapCacheServer to cache the token until expiry but maybe
> there is a better way.
> [https://host:port/nifi-api/access/token] POST
> This endpoint only supports form based authentication but if it supported
> basic authentcation too it would be simpler becasue then the InvokeHttp
> processor could access it and the password would be hidden by sensitive
> property. It doesn't help to actually hide the token in the flow. Howdo you
> store sensitve values at attribute in the flow? Maybe it is wrong thinking to
> do so.
> It seems like many are struggling with oAuth2 token based authentication as
> well. client credentials grant and password credentials grant and so on.
> Havne't found any cookbook for that.
>
>
> I'm on Ubuntu and java 1.11 sap machine
> In Dell Boomi they have [https://ace.c9.io/] as script editor. It's almost a
> bug not to have a decent script editor for InvokeScriptedProcessor and the
> other script processors. The tiny little NiFi script window is painful to
> wotk with and closes when you press enter to create a new line.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
