Jim, You should be able to do everything you want with InvokeScriptedProcessor and a handy utility class in NiFi called SynchronousFileWatcher [1]. It is used by various NiFi components such as the ScanAttribute [2] processor. You should be able to import it via:
from org.apache.nifi.util.file.monitor import SynchronousFileWatcher Check the code/javadoc for usage, you should also be able to port the relevant ScanAttribute code to Jython as well. One caveat is that the Processor interface doesn't have a shutdown/stop method, and InvokeScriptedProcessor does not yet support annotated lifecycle methods (NIFI-2215 [3]), so be aware that there might be cleanup and/or memory leak issues if the ISP is started and stopped alot. However using SynchronousFileWatcher should allow you to keep the ISP running as long as you like. Regards, Matt [1] https://github.com/apache/nifi/blob/master/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/monitor/SynchronousFileWatcher.java [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java#L153 [3] https://issues.apache.org/jira/browse/NIFI-2215 On Wed, Sep 20, 2017 at 8:43 AM, James McMahon <[email protected]> wrote: > Our NiFi is 0.7.x. I understand that it is possible to set customer-specific > configuration parameters in a second config file similar to nifi.properties. > I have this requirement: > > R1- allow engineering leads to set valid message types in a simple python > list, like so: validTypes = ['larry','curly','moe'] > > R2- application specific parms such as the validTypes list should be easily > editable by lead engineers who have no knowledge of NiFi, and so should be > maintained in a simple system file (text or xml, for example) > > R3- after editing this file, a NiFi restart should not be required. Changes > should be auto-detected by a processor in the workflow. > > I'd like to ask three related questions. > > Q1- If I was to set such parms in the second configuration file, how would I > load those values into attributes that would be associated with each > flowfile handled by my workflow? Can you refer me to an example? > > Q2- I'm given to understand that the approach for Q1 would require a restart > of nifi in order to pick up the new values. Is it possible instead to use an > InvokeScriptedProcessor (ISP) in my workflow to do this instead? I have used > ISP to redirect log messages to a log file of my choosing. ISP permits you > to define actions that happen only once at processor start, and so do not > get repeated for each and every flowfile handled by the processor. I set up > logging in this manner now in a python script executed by ISP: > def initialize(self, context): > try: > self.logger = logging.getLogger(....) > self.logger.setLevel(logging.DEBUG) > except: > .... > > How can I set attributes - common attributes - inside that initialize() > method that will be inherited by each flowfile passing through my ISP > instance? > I am hoping this is possible, because then edits to a simple text file > will not require a restart of nifi following changes - I will only have to > restart our ISP. And while this is not entirely "hands free" as required by > #R3, it is preferred to disrupting the entire nifi instance with a restart. > > Q3- How can I incorporate processors in my workflow to auto-detect a change > in a system file, and reload those system file contents to attributes? > > Thanks in advance for your guidance. -Jim
