Hi Matt, Thank you taking the time to explain and provide necessary inputs.
Regards, Harsha Sent from Outlook<http://aka.ms/weboutlook> ________________________________ From: Matt Burgess <[email protected]> Sent: Thursday, February 14, 2019 12:59 PM To: [email protected] Subject: Re: InvokeScriptedProcessor initialize method Harsha, You are correct, the ScriptEngine is supposed to destroy all the objects created by the script when the scripted Processor instance is destroyed (which it should be on stopped). However not all objects "like to be destroyed" and want API methods to be called on them first. This IMHO is a bit of a gap in capability, in a regular processor you can annotate a method with @OnStopped and it will get called when the processor is stopped. However there is no onStopped() or stop() method on the Processor API, to which ISP scripts much currently adhere. There is a Jira case [1] to add support for annotated methods, but since not all scripting engines/languages support Java Annotations, and to be consistent with other added behavior (such as calling setLogger() if it exists in the scripted processor), I'm thinking of changing [1] to call specific methods for annotated events, such as onStopped() and onShutdown(). Then any scripting engine should be able to leverage this capability. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-2215 On Thu, Feb 14, 2019 at 11:56 AM Sri Harsha Chavali <[email protected]> wrote: > > Hi Matt, > > Thank you for the quick response. I took a while to try and understand how > initialize and validate are working. I have tried a few cases and noticed > that: > > Yes the ObjectIds' are different and > Validate only runs periodically when the processor is stopped. When the > processor is running the validate is not called. > First time the ISP is created and script is placed in the body an object is > created and the same object is used for all FlowFiles and when the processor. > When the processor is stopped a new object is created (from within validate) > and validate is run periodically and the script object remains the same. > Second time when I start the processor this previous object (the one I > mentioned in 4th bullet point above) that is created is used for all > flowfiles. > > Logs: > Instantiate- 2019-02-14 11:34:48,005 INFO [NiFi logging handler] > org.apache.nifi.StdOut Inside initialize() method ProcessMessagesTest@1e84d2e2 > Periodical Validation: > 2019-02-14 11:34:48,005 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@1e84d2e2 > 2019-02-14 11:34:49,811 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@1e84d2e2 > 2019-02-14 11:34:54,823 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@1e84d2e2 > 2019-02-14 11:34:59,837 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@1e84d2e2 > > onTrigger: > 2019-02-14 11:37:20,840 INFO [NiFi logging handler] org.apache.nifi.StdOut > Reading Flowfile onTrigger: ProcessMessagesTest@1e84d2e2 > > Stop: > 2019-02-14 11:37:40,641 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside initialize() method ProcessMessagesTest@3bd3682b > 2019-02-14 11:37:40,641 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@3bd3682b > 2019-02-14 11:37:45,659 INFO [NiFi logging handler] org.apache.nifi.StdOut > Inside Validation part of the script with ID: ProcessMessagesTest@3bd3682b > Restart: > 2019-02-14 11:37:53,503 INFO [NiFi logging handler] org.apache.nifi.StdOut > Reading Flowfile onTrigger: ProcessMessagesTest@3bd3682b > 2019-02-14 11:37:53,503 INFO [NiFi logging handler] org.apache.nifi.StdOut > Reading Flowfile onTrigger: ProcessMessagesTest@3bd3682b > > So the initialize is called only once during actual script validation. All > the script objects that are created are destroyed upon processor stop. Please > correct me if I'm wrong. > > One additional question is: > What if I want to initialize a connection to Database or a java client > inside the initialize() method. Is the connection or java client > automatically closed/destroyed upon processor stop? > > Thank you, > Harsha > > Sent from Outlook > > > ________________________________ > From: Matt Burgess <[email protected]> > Sent: Thursday, February 14, 2019 10:15 AM > To: [email protected] > Subject: Re: InvokeScriptedProcessor initialize method > > Harsha, > > This is a feature/bug related to the difference between the lifecycle > events handled by InvokeScriptedProcessor (ISP) itself and the > scripted processor to which some lifecycle events get delegated. > > For normal processors (including ISP), the initialize() method is > called when the processor is instantiated. We can't delegate to the > scripted processor, as the properties are not usually populated at > that time (unless you're instantiating a template maybe). The closest > we can get is when the processor is (re)loaded from the file or Script > Body property. Ideally what would happen is that when ISP is stopped, > we remove the reference to the existing scripted processor (which we > do) and when the processor is started, a new instance would be loaded > from the script, and initialize called on it at that time. However, > asynchronously (and periodically) the ISP processor gets > customValidate() called upon it, and we try to delegate to the > scripted processor's validate() method. To do that we need to load the > script at that time, and then we call initialize on it. So technically > initialize is not being called when the processor is stopped, but it > is being marked as "needs to be reloaded", then whenever > customValidate is called, initialize is called as a result. > > You can see that the two calls to initialize() are on different > instances by changing your output line in your scripted initialize() > method to: > > log.info("Inside initialize() method for ${this.toString()}") > > In logs/nifi-app.log you can see that they have different Object IDs. > > Not to say that this is the "correct" behavior, but it may be more of > a feature than a bug, as initialize does only get called once, albeit > at a possibly awkward time. > > Regards, > Matt > > On Thu, Feb 14, 2019 at 9:00 AM Sri Harsha Chavali > <[email protected]> wrote: > > > > Hi All, > > > > I have a question regarding the behavior I'm noticing in > > InvokeScriptedProcessor. I was testing a functionality and noticed that > > initialize method in InvokeScriptedProcessor is called twice during the > > life cycle of one full run. To reproduce this I kept simple print statement > > in the initialize method and saw the statement being printed twice. > > > > First time when the processor is setup and my custom code is validated > > Second time after one execution is completed and the processor is stopped. > > > > See my sample code attached. I see the below statements being printed: > > > > 2019-02-14 08:48:30,585 INFO [NiFi logging handler] org.apache.nifi.StdOut > > Inside initialize() method > > 2019-02-14 08:48:31,967 INFO [NiFi logging handler] org.apache.nifi.StdOut > > Reading Flowfile onTrigger > > 2019-02-14 08:48:33,959 INFO [NiFi logging handler] org.apache.nifi.StdOut > > Inside initialize() method > > > > Is this expected that initialize method is called when the processor is > > stopped? I was under the assumption that initialize method is only called > > during the processor setup. Please help. > > > > Thank you, > > Harsha > > > > Sent from Outlook
