[ https://issues.apache.org/jira/browse/NIFI-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Burgess resolved NIFI-8684. -------------------------------- Fix Version/s: 1.15.0 Assignee: Matt Burgess Resolution: Fixed Closing as fixed by NIFI-7012 > sensitive property not working for InvokeScriptedProcessor > ---------------------------------------------------------- > > Key: NIFI-8684 > URL: https://issues.apache.org/jira/browse/NIFI-8684 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Affects Versions: 1.13.2 > Reporter: Jul Tomten > Assignee: Matt Burgess > Priority: Major > Labels: InvokeScriptedProcessor, property, sensitive > Fix For: 1.15.0 > > > I use InvokeScriptedProcessor > I'm trying to read a sensitive property from the process context > > before restarting NiFi it was working fine > after restarting NiFi - NiFi fails to startup with the error below > see https://issues.apache.org/jira/browse/NIFI-7012 > > > 2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer > Failed to start web server... shutting down. > org.apache.nifi.controller.serialization.FlowSynchronizationException: > java.lang.IllegalArgumentException: The property 'Password3' cannot reference > Parameter 'password3' because Sensitive Parameters may only be referenced by > Sensitive Properties. > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306) > at > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89) > at > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810) > at > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539) > at > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) > at > org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997) > at > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911) > at > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) > at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) > at org.eclipse.jetty.server.Server.start(Server.java:423) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) > at org.eclipse.jetty.server.Server.doStart(Server.java:387) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057) > at org.apache.nifi.NiFi.<init>(NiFi.java:159) > at org.apache.nifi.NiFi.<init>(NiFi.java:71) > at org.apache.nifi.NiFi.main(NiFi.java:303) > Caused by: java.lang.IllegalArgumentException: The property 'Password3' > cannot reference Parameter 'password3' because Sensitive Parameters may only > be referenced by Sensitive Properties. > at > org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313) > at > org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198) > at > org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61) > at > org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333) > at > org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322) > at > org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432) > at > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283) > ... 43 common frames omitted > > in flow.xml.gz > > > {code:java} > <parameterContext> > <id>f8098456-0179-1000-40b0-ba2fcee07a72</id> > <name>otto_test</name> > <description/> > <parameter> > <name>password</name> > <description/> > <sensitive>true</sensitive> > <value>enc > {61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230} > </value> > </parameter> > <parameter> > <name>password3</name> > <description/> > <sensitive>true</sensitive> > <value>enc > {6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b} > </value> > </parameter> > </parameterContext> > {code} > > > > > > > {code:java} > JIRA messed up and inserted lots of brackets {{ and }} all over the place, I > tried to remove them but may have not got them all or to many removed{code} > {code:java} > <processor> > <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id> > <name>InvokeScriptedProcessor2</name> > <position x="-1144.0" y="3640.0"/> > <styles/> > <comment/> > > <class&gt;org.apache.nifi.processors.script.InvokeScriptedProcessor</class&gt; > <bundle> > <group>org.apache.nifi</group> > <artifact>nifi-scripting-nar</artifact> > <version>1.13.2</version> > </bundle> > <maxConcurrentTasks>1</maxConcurrentTasks> > <schedulingPeriod>0 sec</schedulingPeriod> > <penalizationPeriod>30 sec</penalizationPeriod> > <yieldPeriod>1 sec</yieldPeriod> > <bulletinLevel>WARN</bulletinLevel> > <lossTolerant>false</lossTolerant> > <scheduledState>RUNNING</scheduledState> > <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy> > <executionNode>ALL</executionNode> > <runDurationNanos>0</runDurationNanos> > <property> > <name>Script Engine</name> > <value>ECMAScript</value> > </property> > <property> > <name>Script File</name> > </property> > <property> > <name>Script Body</name> > > <value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html > > //http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html > > //http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html > > //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html > > //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html > > //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html > > //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html > /////////////////////////////////////////////////////////// > // "imports" go here > //////////////////////////////////////////////////////////// > var StreamCallback = > Java.type("org.apache.nifi.processor.io.StreamCallback"); > var IOUtils = Java.type("org.apache.commons.io.IOUtils"); > var StandardCharsets = Java.type("java.nio.charset.StandardCharsets"); > var Processor = Java.type("org.apache.nifi.processor.Processor"); > var Relationship = Java.type("org.apache.nifi.processor.Relationship"); > var StandardValidators = > Java.type("org.apache.nifi.processor.util.StandardValidators"); > var Validator = Java.type("org.apache.nifi.components.Validator"); > > var HashSet = Java.type("java.util.HashSet"); > var LinkedList = Java.type("java.util.LinkedList"); > var PropertyDescriptorBuilder = > Java.type("org.apache.nifi.components.PropertyDescriptor.Builder"); > var log = null; > var REL_SUCCESS = new > Relationship.Builder().name("success").description('FlowFiles that were > successfully processed are routed here').build(); > var REL_FAILURE = new > Relationship.Builder().name("failure").description('FlowFiles that were not > successfully processed are routed here').build(); > var strPassword2 = "Password2"; > var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2) > .description("Password used to connect") > .required(true) > //.sensitive(false) > //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) > .expressionLanguageSupported(true) > .addValidator(Validator.VALID) > .build(); > var strPassword3 = "Password3"; > var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3) > .description("Password used to connect") > .required(true) > .sensitive(true)}} > // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) > .expressionLanguageSupported(false) > .addValidator(Validator.VALID) > .build(); > function executeScript(session, context, log, REL_SUCCESS, REL_FAILURE) { > //////////////////////////////////////////////////////////// > // your code goes here > //////////////////////////////////////////////////////////// > var pwd2 = context.getProperty(strPassword2).getValue(); > var pwd3 = context.getProperty(strPassword3).getValue(); > var pwd31 = > context.getProperty(strPassword3).evaluateAttributeExpressions().getValue(); > log.error('pwd2:'+ pwd2); > log.error('pwd3:'+ pwd3); > var flowFile = session.get(); > if(flowFile != null) {}}if(flowFile != null) { > // Create a new StreamCallback, passing in a function to define the > interface method > flowFile = session.write(flowFile, > new StreamCallback(function(inputStream, outputStream){ > var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8); > outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8)); > > outputStream.write("\npwd2:".getBytes()); > outputStream.write(pwd2.getBytes()); > // outputStream.write("\npwd3:".getBytes()); > // outputStream.write(pwd3.getBytes()); > // outputStream.write("\n31:".getBytes()); > // outputStream.write(pwd31.getBytes()); > outputStream.write("\n".getBytes()); } > )); > > try{ > session.transfer(flowFile, REL_SUCCESS) > }catch(e){ > log.error('Something went wrong', e) > session.transfer(flowFile, REL_FAILURE) > } > } > function initialize(context){ log = context.logger; } > function getRelationships() > { > var r = new HashSet(); > r.add(REL_FAILURE); > r.add(REL_SUCCESS); > return r; > } > function validate(context){ return null; } > function getPropertyDescriptor(name) { > if(name.equals(strPassword2)){ > return prop_password2; > } else if(name.equals(strPassword3)){ > return prop_password3; > } else{{ > return null; > } > } > function onPropertyModified(descriptor, oldValue, newValue) { return null; } > function getPropertyDescriptors(){ > var r = new LinkedList(); > r.add(prop_password2); > r.add(prop_password3); > return r; > } > function getIdentifier(){ return null; } > function onTrigger(context, sessionFactory) { > var session = sessionFactory.createSession(); > try{ > executeScript(session, context, log, REL_SUCCESS, REL_FAILURE); > session.commit(); > }catch (t) { > log.error("{} failed to process due to {}; rolling back session", > Java.to([this, t], "java.lang.Object[]")); > session.rollback(true); > throw t; > } > } > processor = this;</value> > </property> > <property> > <name>Module Directory</name> > </property> > <property> > <name>Password2</name> > <value>xyz</value> > </property> > <property> > <name>Password3</name> > <value>enc > {da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da} > </value> > </property> > </processor> > > {code} > > to recover I had to uncomment the code and remove the property: Password3 > using a text editor on the flow.xml > > Many thanks to mburgess the cookbook author. > I'm new to NiFi so sorry if I've simply made a programming error. > It seems like I'm not the only one with the similar problem. > I try to createa flow that login to provenance repository in a NiFi with LDAP > using the InvokeHttp in a later step. To login I need to fetch a JWT-token > and then call the provenanc repo REST apis using the token. When fetching the > token I want to hide tha password. Maybe there is a better way. I also would > like to store the token in a secure way and not store it as attribute. I use > the DistributedMapCacheServer to cache the token until expiry but maybe > there is a better way. > [https://host:port/nifi-api/access/token] POST > This endpoint only supports form based authentication but if it supported > basic authentcation too it would be simpler becasue then the InvokeHttp > processor could access it and the password would be hidden by sensitive > property. It doesn't help to actually hide the token in the flow. Howdo you > store sensitve values at attribute in the flow? Maybe it is wrong thinking to > do so. > It seems like many are struggling with oAuth2 token based authentication as > well. client credentials grant and password credentials grant and so on. > Havne't found any cookbook for that. > > > I'm on Ubuntu and java 1.11 sap machine > In Dell Boomi they have [https://ace.c9.io/] as script editor. It's almost a > bug not to have a decent script editor for InvokeScriptedProcessor and the > other script processors. The tiny little NiFi script window is painful to > wotk with and closes when you press enter to create a new line. > -- This message was sent by Atlassian Jira (v8.3.4#803005)