Hi Selvam,

Supposing that your source if a SQL-like source, you should have a look at
QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns'
that gives you the possibility to specify the column containing an ID and
or timestamp. The processor will keep track of the maximum value for each
column that has been returned since the processor started running. This can
be used to retrieve only those rows that have been added/updated since the
last retrieval. Note that some JDBC types such as bit/boolean are not
conducive to maintaining maximum value, so columns of these types should
not be listed in this property, and will result in error(s) during
processing. If no columns are provided, all rows from the table will be
considered, which could have a performance impact.

[1] https://nifi.apache.org/docs/nifi-docs/


2016-09-20 8:50 GMT+02:00 Selvam Raman <sel...@gmail.com>:

> > Hi,
> >
> > We have a requirement to pull data periodically from a data source. For
> this to work we would like nifi to keep track of the last id or time stamp
> that was pulled successfully so that the next pull starts from that point.
> >
> > Let me know if nifi supports this? If yes how do we configure?
> >
> > --
> > Selvam Raman
> > "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"

Reply via email to