Jim, You need to call session.commit() periodically if you want to progressively push flowfiles out. Though you need to think about the failure scenarios your script has and whether you really want to sacrifice the ability to do a rollback in the event things go wrong.
On Thu, Feb 15, 2018 at 6:27 AM, James McMahon <[email protected]> wrote: > Matt, I have noticed another behavior pattern I hope to correct, but I > don't fully understand it and so hope that you can provide some insights. > > I have embedded the flowFile related commands to create the flowFiles > within my file processing loop. My expectation was that I would see the > output queue from ExecuteScript increment as each file was created. But I > don't see that behavior. instead, I see a zero in the output queue for > several minutes, and then suddenly the count jumps to the total set of > flowFiles created. > > Because working back through the directory structure in time can be a time > consuming process, I hope to output to flow incremental results as files > are found. How can I make that happen? Is there a means to periodically > flush what is in the output buffer every N files, perhaps? > > Thanks again for your insights. -Jim > > On Mon, Feb 12, 2018 at 12:24 PM, James McMahon <[email protected]> > wrote: > >> I see. Thank you Matt. -Jim >> >> On Mon, Feb 12, 2018 at 12:10 PM, Matt Burgess <[email protected]> >> wrote: >> >>> Jim, >>> >>> In this case I don't think it's as much that the modules aren't being >>> found, rather that the datetime module in Jython returns >>> java.sql.Timestamp (Java) objects, rather than Jython/Python datetime >>> objects, and the former do not support the methods/attributes of the >>> latter, including timetuple(). Apparently [1] this change was made >>> around Jython 2.5, and NiFi uses 2.7.1. >>> >>> Looks like you'll need to write your own timetuple() function, using >>> the java.sql.Timestamp [2] and related APIs. >>> >>> Regards, >>> Matt >>> >>> [1] http://www.jython.org/javadoc/com/ziclix/python/sql/Jython22 >>> DataHandler.html >>> [2] https://docs.oracle.com/javase/8/docs/api/java/sql/Timestamp.html >>> >>> On Mon, Feb 12, 2018 at 11:58 AM, James McMahon <[email protected]> >>> wrote: >>> > Good afternoon. I havd a python script that I can execute from the >>> command >>> > line via my python interpreter. In it, I do this >>> > >>> > myTime = time.mktime(myDateTime.timetuple()) >>> > >>> > When I try to run from my ExecuteScript processor in NiFi, this is not >>> > recognized. This error gets thrown: >>> > >>> > 'java.sql.Timestamp' object has no attribute 'timetuple' in <script> >>> > >>> > Why would the available python modules not be found in the >>> ExecuteScript yet >>> > are found outside NiFi? >>> > >>> > Thanks very much. -Jim >>> >> >> >
