Hi Selvam, Supposing that your source if a SQL-like source, you should have a look at QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns' that gives you the possibility to specify the column containing an ID and or timestamp. The processor will keep track of the maximum value for each column that has been returned since the processor started running. This can be used to retrieve only those rows that have been added/updated since the last retrieval. Note that some JDBC types such as bit/boolean are not conducive to maintaining maximum value, so columns of these types should not be listed in this property, and will result in error(s) during processing. If no columns are provided, all rows from the table will be considered, which could have a performance impact.
[1] https://nifi.apache.org/docs/nifi-docs/ Pierre 2016-09-20 8:50 GMT+02:00 Selvam Raman <sel...@gmail.com>: > > > Hi, > > > > We have a requirement to pull data periodically from a data source. For > this to work we would like nifi to keep track of the last id or time stamp > that was pulled successfully so that the next pull starts from that point. > > > > Let me know if nifi supports this? If yes how do we configure? > > > > -- > > Selvam Raman > > "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து" >
