Andy, thank you very much. This is a section of my code excerpted and hand
jammed in here by me to give the question context. I apologize: I
introduced those typos when hand jamming. They are not in the code, but
thanks for pointing them out here. I do have the correct imports.
Unfortunately I have to depend on a PyStreamCallback() class, and so can't
employ a direct api call such as WriteContentCallback("Hello there this is
my data”) in my code.
>From what I am seeing, logging to a rotating log file from an ExecuteScript
results in odd multiple writes to the log that increase in number with each
subsequent flowfile. I suspect it has to do with the way the log handle is
not closed properly.
I gather from some research that the way to do this is to employ an
InvokeScriptedProcessor (ISP) - not ExecuteScript. Problem is, I can't find
an example that shows how the PyStreamCallback is integrated with the
processor class in an ISP. Matt Burgess speaks to it here
http://funnifi.blogspot.com/2017/11/invokescriptedprocessor-template.html
but this example doesn't show how to integrate the PyStreamCallback. It is
typically a class of its own in the python ExecuteScript python script, and
I don't see how I can fold that into the def executeScript() in this
section of Matt's example:
class E():
def __init__(self):
pass
def executeScript(self,session, context, log, REL_SUCCESS, REL_FAILURE):
#////////////////////////////////////////////////////////////
#// Replace 'pass' with your code
#////////////////////////////////////////////////////////////
pass
#end class
Do you by chance know of a more concrete example for Python,
InvokeScriptedProcessor, rotating log, PyStreamCallback included? Perhaps
even just offer a simple example of the way the execute script and the
callback would be structured in the executeScript method shown above?
Jim
On Thu, Feb 21, 2019 at 9:52 PM Andy LoPresto <[email protected]> wrote:
> Jim,
>
> I had started to provide advice on configuring the logback.xml file when I
> realized you want to output log messages directly to your own log file, not
> using the provided log interface that ExecuteScript offers. I don’t know
> Python well enough to be sure, but I get an error trying to execute your
> script because “logging.handlers.RotatingFileHandler…” can’t be found. You
> may need to configure your processor with a specific module in the modules
> directory, or import that class explicitly. I also want to note there is an
> unclosed single quote on line 11 [ incoming =
> flowFile.getAttribute(‘filename) ] which may be causing some problems — I
> had to close it before I could detect the actual issue. I also had to
> change line 13 to [ flowFile = session.write(flowFile,
> WriteContentCallback("Hello there this is my data”)) ] in order to run this
> example locally.
>
> Andy LoPresto
> [email protected]
> *[email protected] <[email protected]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 21, 2019, at 12:20 PM, James McMahon <[email protected]> wrote:
>
> We are running NiFi 1.8, trying to log to a rotating log file from a
> python script executed by an ExecuteScript processor. We are seeing no
> output to the log file.
> I seem to be establishing the log file handler and the logger without any
> errors.
> I see the log file in my log output directory, so I know it gets created.
> Can this approach be used from Execute Script? What am i neglecting?
>
> My approach is as follows:
>
> flowFiles = session.get(100)
> for flowFile in flowFiles :
> if (flowFile != None) :
> fh =
> logging.handlers.RotatingFileHandler('/opt/nifi/logs/myLog.log',
> maxBytes=500000, backupCount=20)
> fh.setLevel(logging.DEBUG)
> formatter = logging.Formatter('%(asctime)s %(name)s
> %(levelname)s %(message)s')
> fh.setFormatter(formatter)
> logger = logging.getLogger()
> logger.addHandler(fh)
>
> incoming = flowFile.getAttribute('filename)
> logger.info('about to process: %s', incoming)
> flowFile = session.write(flowFile, PyStreamCallback()) # heavy
> lifting in this PyStreamCallback
> logger.info('completed processing: %s', incoming)
>
> fh.close()
> while logger.handlers :
> logger.handlers.pop()
> del fh
>
> session.transfer(flowFile, REL_SUCCESS)
>
> Thanks in advance for your help. -Jim
>
>
>
