Giovanni, That should indeed a good way to get better performance from the processor (the amount of improvement depends on which ScriptEngine you're using), although I'm not sure if I'm the one who pointed it out ;)
The constant (1000) itself could be defined by a dynamic property, whose name becomes the variable passed into the script's context, and its PropertyValue object as the value, just call getValue() to use it as-is, or you can support Expression Language too, check out my ExecuteScript Cookbook Part 3 [1] :) Regards, Matt [1] https://community.hortonworks.com/articles/77739/executescript-cookbook-part-3.html On Sat, May 6, 2017 at 4:29 AM, Giovanni Lanzani <[email protected]> wrote: > Please do not remove the Python scripting facilities. > > I believe most people experience it as very slow because (ok, Python is slow) > they only get a flow file at a time. > > I think Matt pointed out in this ML once that you can use > > flowfiles = session.get(1000) > for flowfile in filter(None, flowfiles): > # do things > > In that case Jython will be kept alive much longer. Or am I missing something? > > Giovanni > >> -----Original Message----- >> From: Joe Witt [mailto:[email protected]] >> Sent: Saturday, May 6, 2017 1:12 AM >> To: [email protected] >> Subject: Re: Is it possible to reference python requests module in >> ExecuteScript? >> >> It is worth discussing whether there is sufficient value to warrant keeping >> jython/python support in the processors or whether we should pull it. It is >> certainly something we can document as being highly limited but we don't >> really know how limited. Frankly given the performance I've seen with it >> I'd be >> ok removing it entirely. One is better off calling the script via a system >> call. >> Groovy is one that I've seen perform very well and be fully featured. >> >> On Fri, May 5, 2017 at 6:38 PM, Russell Bateman <[email protected]> >> wrote: >> > We really want to use ExecuteScript because our end users are Pythonistas. >> > They tend to punctuate their flows with the equivalent of PutFile and >> > GetFile with Python scripts doing stuff on flowfiles that pass out of >> > NiFi before returning into NiFi. >> > >> > However, we find it nearly impossible to replace even the tamest of >> > undertakings. If there were a good set of NiFi/Python shims that, from >> > PyCharm, etc., gave us the ability to prototype, test and debug before >> > copying and pasting into ExecuteScript, that would be wonderful. It >> > hasn't worked out that way. Most of our experience is copying, pasting >> > into the processor property, only to find something wrong, sometimes >> > syntax, sometimes something runtime. >> > >> > On their behalf, I played with this processor a few hours a while back. >> > Another colleague too. Googling this underused tool hasn't been >> > helpful, so the overall experience is negative so far. I can get most >> > of the examples out there to work, but as soon as I try to do "real" >> > work from my point of view, my plans sort of cave in. >> > >> > Likely the Groovy and/or Ruby options are better? But, we're not >> > Groovy or Ruby guys here. I understand the problems with this tool and >> > so I understand what the obstacles are to it growing stronger. The >> > problems won't yield to a few hours one Saturday afternoon. Better >> > problem-logging underneath and >> > better- and more lenient Python support on top. The second one is >> > tough, though. >> > >> > My approach is to minimize those black holes these guys put into their >> > flows by creating custom processors for what I can't solve using >> > standard processors. >> > >> > Trying not to be too negative here... >> > >> > >> > On 05/05/2017 04:09 PM, Andre wrote: >> > >> > Mike, >> > >> > I believe it is possible to use requests under jython, however the >> > process isn't very intuitive. >> > >> > I know one folk that if I recall correctly has used it. Happy to try >> > to find out how it is done. >> > >> > Cheers >> > >> > On Sat, May 6, 2017 at 4:57 AM, Mike Harding <[email protected]> >> wrote: >> >> >> >> Hi All, I'm now looking at using ExecuteScript and python engine to >> >> execute HTTP requests using the requests module. I've tried >> >> referencing requests the module but when I try to import requests I >> >> get a module reference error. >> >> I downloaded the module from here > >> >> https://pypi.python.org/pypi/requests >> >> Not sure why it isnt picking it up. Ive tried referencing the >> >> directory and the .py directly with no success. >> >> Any ideas where im going wrong? >> >> Cheers, >> >> Mike
