There are incremental data movement processors in NiFi depending on
your source/target. For example, if your sources are files, you can
use ListFile in combination with FetchFile, the former will keep track
of which files it has found thus far, so if you put new files into the
location (or update existing ones), only those new/updated files will
be processed the next time.
For database (RDBMS) sources, there are the QueryDatabaseTable and
GenerateTableFetch processors, which support the idea of "maximum
value columns", such that for each of said columns, the processor(s)
will keep track of the maximum value observed in that column, then for
future executions of the processor, only rows whose values in those
columns exceed the currently-observed maximum will be retrieved, then
the maximum will be updated, and so forth.
The Usage documentation for these processors can be found at
https://nifi.apache.org/docs.html (left-hand side under Processors).
On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr <gop....@gmail.com> wrote:
> Hi All,
> I am learning NiFi as well as trying to deploy it in production for few use
> cases. One of the use case is ETL and another use case is, using NiFi as a
> backup solution, where it takes the data from one source and moves to
> another database|file. Is anyone using NiFi for this purpose? Does NiFi
> support incremental data move?
> It would be awesome if someone can point me to right documentation.