Sounds very promising. Thank you very much for pointing me in the right direction Matt. I will start digging into this immediately, and if I'm able to get it to work will follow-up later to post my solution. --Jim
On Wed, Sep 20, 2017 at 10:04 AM, Matt Burgess <[email protected]> wrote: > Jim, > > You should be able to do everything you want with > InvokeScriptedProcessor and a handy utility class in NiFi called > SynchronousFileWatcher [1]. It is used by various NiFi components > such as the ScanAttribute [2] processor. You should be able to import > it via: > > from org.apache.nifi.util.file.monitor import SynchronousFileWatcher > > Check the code/javadoc for usage, you should also be able to port the > relevant ScanAttribute code to Jython as well. One caveat is that the > Processor interface doesn't have a shutdown/stop method, and > InvokeScriptedProcessor does not yet support annotated lifecycle > methods (NIFI-2215 [3]), so be aware that there might be cleanup > and/or memory leak issues if the ISP is started and stopped alot. > However using SynchronousFileWatcher should allow you to keep the ISP > running as long as you like. > > Regards, > Matt > > [1] https://github.com/apache/nifi/blob/master/nifi-commons/ > nifi-utils/src/main/java/org/apache/nifi/util/file/monitor/ > SynchronousFileWatcher.java > [2] https://github.com/apache/nifi/blob/master/nifi-nar- > bundles/nifi-standard-bundle/nifi-standard-processors/src/ > main/java/org/apache/nifi/processors/standard/ScanAttribute.java#L153 > [3] https://issues.apache.org/jira/browse/NIFI-2215 > > On Wed, Sep 20, 2017 at 8:43 AM, James McMahon <[email protected]> > wrote: > > Our NiFi is 0.7.x. I understand that it is possible to set > customer-specific > > configuration parameters in a second config file similar to > nifi.properties. > > I have this requirement: > > > > R1- allow engineering leads to set valid message types in a simple python > > list, like so: validTypes = ['larry','curly','moe'] > > > > R2- application specific parms such as the validTypes list should be > easily > > editable by lead engineers who have no knowledge of NiFi, and so should > be > > maintained in a simple system file (text or xml, for example) > > > > R3- after editing this file, a NiFi restart should not be required. > Changes > > should be auto-detected by a processor in the workflow. > > > > I'd like to ask three related questions. > > > > Q1- If I was to set such parms in the second configuration file, how > would I > > load those values into attributes that would be associated with each > > flowfile handled by my workflow? Can you refer me to an example? > > > > Q2- I'm given to understand that the approach for Q1 would require a > restart > > of nifi in order to pick up the new values. Is it possible instead to > use an > > InvokeScriptedProcessor (ISP) in my workflow to do this instead? I have > used > > ISP to redirect log messages to a log file of my choosing. ISP permits > you > > to define actions that happen only once at processor start, and so do not > > get repeated for each and every flowfile handled by the processor. I set > up > > logging in this manner now in a python script executed by ISP: > > def initialize(self, context): > > try: > > self.logger = logging.getLogger(....) > > self.logger.setLevel(logging.DEBUG) > > except: > > .... > > > > How can I set attributes - common attributes - inside that initialize() > > method that will be inherited by each flowfile passing through my ISP > > instance? > > I am hoping this is possible, because then edits to a simple text > file > > will not require a restart of nifi following changes - I will only have > to > > restart our ISP. And while this is not entirely "hands free" as required > by > > #R3, it is preferred to disrupting the entire nifi instance with a > restart. > > > > Q3- How can I incorporate processors in my workflow to auto-detect a > change > > in a system file, and reload those system file contents to attributes? > > > > Thanks in advance for your guidance. -Jim >
