Thanks Joe and Matt. @Joe, based on your comment, I need to use NiFi as a producer which puts the data on Kafka queue and then have NiFi consumer, which writes the data back to the destination. Is my understanding correct?
@Matt, My use case is for the DynamoDB. I will look into whether incremental copy is supported for Dynamodb. Thanks again and felt so good to see the vibrant community. I got my questions answered within five minutes. Kudos to NiFi community. On Thu, Oct 13, 2016 at 8:17 AM, Matt Burgess <mattyb...@apache.org> wrote: > Rai, > > There are incremental data movement processors in NiFi depending on > your source/target. For example, if your sources are files, you can > use ListFile in combination with FetchFile, the former will keep track > of which files it has found thus far, so if you put new files into the > location (or update existing ones), only those new/updated files will > be processed the next time. > > For database (RDBMS) sources, there are the QueryDatabaseTable and > GenerateTableFetch processors, which support the idea of "maximum > value columns", such that for each of said columns, the processor(s) > will keep track of the maximum value observed in that column, then for > future executions of the processor, only rows whose values in those > columns exceed the currently-observed maximum will be retrieved, then > the maximum will be updated, and so forth. > > The Usage documentation for these processors can be found at > https://nifi.apache.org/docs.html (left-hand side under Processors). > > Regards, > Matt > > On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr <gop....@gmail.com> wrote: > > Hi All, > > I am learning NiFi as well as trying to deploy it in production for few > use > > cases. One of the use case is ETL and another use case is, using NiFi as > a > > backup solution, where it takes the data from one source and moves to > > another database|file. Is anyone using NiFi for this purpose? Does NiFi > > support incremental data move? > > It would be awesome if someone can point me to right documentation. > > Thanks > > Rai >
