Thanks for sharing the solution Mike. Is there something we need to update in nifi to prevent this from biting others?
Thanks Mark Sent from my iPhone On Sep 2, 2023, at 9:48 AM, Joe Witt <[email protected]> wrote: Nice. Gald you found it. On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen <[email protected]<mailto:[email protected]>> wrote: It was the PostgreSQL JDBC driver. If you don't paginate the query aggressively, it will try to load a significant chunk of the table into memory rather than just pulling chunks, even with fetchSize set low. On Fri, Sep 1, 2023 at 6:01 PM Mike Thomsen <[email protected]<mailto:[email protected]>> wrote: I have a three node cluster with an executesqlrecord processor with primary execution only. The sql it runs is a straight forward select on a table with about 44m records. If I leave it running, after about 10 min the node becomes unresponsive and leaves the cluster. The query runs just fine in jetbrains data grip on that postgresql server, so I don’t think it’s anything weird with the db or query. Any ideas about what could be causing this? Even with a high limit like 5m records the query doesn’t lock up the NiFi node. Sent from my iPhone
