I’m having an issue with the GenerateTableFetch Processor, and I wanted to ask for some insight into whether this is a bug or expected behavior. Using NiFi 1.12.1 I have a MySQL table with 1M+ rows, and I have a GenerateTableFetch processor with a `maximum-value column` and `partition-size` set to 25000 and a `run schedule` of 9 minutes. When the etl starts up I get a sequence of queries for the existing 1M+ rows like this example `SELECT … ORDER BY maxvalcolumn LIMIT 25000 OFFSET 375000`.
The on 9 minutes intervals I get queries like `SELECT … FROM ... WHERE maxvalcolumn > … AND maxvalcolumn <= … ORDER BY maxvalcolumn LIMIT 25000` The issue is that I see only 1 query per 9 minutes with a `LIMIT 25000`, so if my table accumulates more than 25000 rows in 9 minutes the `LIMIT 25000` term simply drops the additional rows and they are passed up. Does the GenerateTableFetch delta copy generate any additional queries with the `OFFSET` term? I’m not sure if there’s a configuration where I can get multiple queries using the `OFFSET` term in the 9 minute interval, or if I can have the query generated without the `LIMIT 25000` term. Thanks, John -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
