[
https://issues.apache.org/jira/browse/NIFI-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jul Tomten updated NIFI-8684:
-----------------------------
Affects Version/s: 1.13.2
> sensitive property not working for InvokeScriptedProcessor
> ----------------------------------------------------------
>
> Key: NIFI-8684
> URL: https://issues.apache.org/jira/browse/NIFI-8684
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.13.2
> Reporter: Jul Tomten
> Priority: Major
>
> I use InvokeScriptedProcessor
> I'm trying to read a sensitive property from the process context
>
> before restarting NiFi it was working fine
> after restarting NiFi - NiFi fails to startup with the error below
>
>
> 2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.IllegalArgumentException: The property 'Password3' cannot reference
> Parameter 'password3' because Sensitive Parameters may only be referenced by
> Sensitive Properties.
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
> at
> org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at org.eclipse.jetty.server.Server.start(Server.java:423)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
> at org.eclipse.jetty.server.Server.doStart(Server.java:387)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
> at org.apache.nifi.NiFi.<init>(NiFi.java:159)
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> at org.apache.nifi.NiFi.main(NiFi.java:303)
> Caused by: java.lang.IllegalArgumentException: The property 'Password3'
> cannot reference Parameter 'password3' because Sensitive Parameters may only
> be referenced by Sensitive Properties.
> at
> org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
> at
> org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
> at
> org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
> ... 43 common frames omitted
>
> in flow.xml.gz
>
> <parameterContext>
> <id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
> <name>otto_test</name>
> <description/>
> <parameter>
> <name>password</name>
> <description/>
> <sensitive>true</sensitive>
>
> <value>enc\{61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}</value>
> </parameter>
> <parameter>
> <name>password3</name>
> <description/>
> <sensitive>true</sensitive>
>
> <value>enc\{6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}</value>
> </parameter>
> </parameterContext>
>
>
>
>
> <processor>
> <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
> <name>InvokeScriptedProcessor2</name>
> <position x="-1144.0" y="3640.0"/>
> <styles/>
> <comment/>
> <class>org.apache.nifi.processors.script.InvokeScriptedProcessor</class>
> <bundle>
> <group>org.apache.nifi</group>
> <artifact>nifi-scripting-nar</artifact>
> <version>1.13.2</version>
> </bundle>
> <maxConcurrentTasks>1</maxConcurrentTasks>
> <schedulingPeriod>0 sec</schedulingPeriod>
> <penalizationPeriod>30 sec</penalizationPeriod>
> <yieldPeriod>1 sec</yieldPeriod>
> <bulletinLevel>WARN</bulletinLevel>
> <lossTolerant>false</lossTolerant>
> <scheduledState>RUNNING</scheduledState>
> <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
> <executionNode>ALL</executionNode>
> <runDurationNanos>0</runDurationNanos>
> <property>
> <name>Script Engine</name>
> <value>ECMAScript</value>
> </property>
> <property>
> <name>Script File</name>
> </property>
> <property>
> <name>Script Body</name>
>
> <value>//[http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html]
>
> //[http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html]
>
> //[http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html]
>
> //[https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html]
>
> //[https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html]
>
> //[https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html]
>
> //[https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html]
> ///////////////////////////////////////////////////////////
> // "imports" go here
> ////////////////////////////////////////////////////////////
> var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
> var IOUtils = Java.type("org.apache.commons.io.IOUtils");
> var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
> var Processor = Java.type("org.apache.nifi.processor.Processor");
> var Relationship = Java.type("org.apache.nifi.processor.Relationship");
> var StandardValidators =
> Java.type("org.apache.nifi.processor.util.StandardValidators");
> var Validator = Java.type("org.apache.nifi.components.Validator");
> org.apache.nifi.components
> var HashSet = Java.type("java.util.HashSet");
> var LinkedList = Java.type("java.util.LinkedList");
> var PropertyDescriptorBuilder =
> Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
> var log = null;
> var REL_SUCCESS = new
> Relationship.Builder().name("success").description('FlowFiles that were
> successfully processed are routed here').build();
> var REL_FAILURE = new
> Relationship.Builder().name("failure").description('FlowFiles that were not
> successfully processed are routed here').build();
> var strPassword2 = "Password2";
> var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
> .description("Password used to connect")
> .required(true)
> //.sensitive(false)
> //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(true)
> .addValidator(Validator.VALID)
> .build();
> var strPassword3 = "Password3";
> var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
> .description("Password used to connect")
> .required(true)
> .sensitive(true)
> // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .expressionLanguageSupported(false)
> .addValidator(Validator.VALID)
> .build();
> function executeScript(session, context, log, REL_SUCCESS, REL_FAILURE) {
> ////////////////////////////////////////////////////////////
> // your code goes here
> ////////////////////////////////////////////////////////////
> var pwd2 = context.getProperty(strPassword2).getValue();
> var pwd3 = context.getProperty(strPassword3).getValue();
> var pwd31 =
> context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
> log.error('pwd2:'+ pwd2);
> log.error('pwd3:'+ pwd3);
> var flowFile = session.get();
> if(flowFile != null) {
> if(flowFile != null) {
> // Create a new StreamCallback, passing in a function to define the
> interface method
> flowFile = session.write(flowFile,
> new StreamCallback(function(inputStream, outputStream)
> { var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
> outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8));
> outputStream.write("\npwd2:".getBytes());
> outputStream.write(pwd2.getBytes()); //
> outputStream.write("\npwd3:".getBytes()); //
> outputStream.write(pwd3.getBytes()); //
> outputStream.write("\n31:".getBytes()); //
> outputStream.write(pwd31.getBytes()); outputStream.write("\n".getBytes()); }
> ));
> }
> try
> { session.transfer(flowFile, REL_SUCCESS) }
> catch(e)
> { log.error('Something went wrong', e) session.transfer(flowFile,
> REL_FAILURE) }
> }
> function initialize(context) \{ log = context.logger; }
> function getRelationships()
> { var r = new HashSet(); r.add(REL_FAILURE); r.add(REL_SUCCESS); return r; }
> function validate(context) \{ return null; }
> function getPropertyDescriptor(name) {
> if(name.equals(strPassword2))
> { return prop_password2; }
> else if(name.equals(strPassword3))
> { return prop_password3; }
> else
> { return null; }
> }
> function onPropertyModified(descriptor, oldValue, newValue) \{ return null; }
> function getPropertyDescriptors()
> { var r = new LinkedList(); r.add(prop_password2); r.add(prop_password3);
> return r; //return null; }
> function getIdentifier() \{ return null; }
> function onTrigger(context, sessionFactory) {
> var session = sessionFactory.createSession();
> try
> { executeScript(session, context, log, REL_SUCCESS, REL_FAILURE);
> session.commit(); }
> catch (t) {
> log.error("{} failed to process due to {}; rolling back session",
> Java.to([this, t], "java.lang.Object[]"));
> session.rollback(true);
> throw t;
> }
> }
> processor = this;</value>
> </property>
> <property>
> <name>Module Directory</name>
> </property>
> <property>
> <name>Password2</name>
> <value>xyz</value>
> </property>
> <property>
> <name>Password3</name>
>
> <value>enc\{da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}</value>
> </property>
> </processor>
>
>
> Many thanks to mburgess the cookbook author.
> I'm new to NiFi so sorry if I've simply made a programming error.
> It seems like I'm not the only one with the similar problem.
> I try to createa flow that login to provenance repository in a NiFi with LDAP
> using the InvokeHttp in a later step. To login I need to fetch a token and
> then call the call the provenanc repo REST apis. When fetching the token I
> want to hide tha password. Maybe there is a better way. I also would like to
> store the token in a secure way and not store it as attribute.
> [https://host:port/nifi-api/access/token] POST
> This endpoint only supports form based authentication but if it supported
> basic authentcation too it would be simpler becasue then the InvokeHttp
> processor could access it and the password would be hidden by sensitive
> property. It doesn't help to actually hide the token in the flow. Howdo you
> store sensitve values at attribute in the flow? Maybe it is wrong thinking to
> do so.
> It seems like many are struggling with oAuth2 token based authentication as
> well. client credentials grant and password credentials grant and so on.
> Havne't found any cookbook for that.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
