Thanks Joe and Matt.
@Joe, based on your comment, I need to use NiFi as a producer which puts
the data on Kafka queue and then have NiFi consumer, which writes the data
back to the destination. Is my understanding correct?
@Matt, My use case is for the DynamoDB. I will look into whether
incremental copy is supported for Dynamodb.
Thanks again and felt so good to see the vibrant community. I got my
questions answered within five minutes. Kudos to NiFi community.
On Thu, Oct 13, 2016 at 8:17 AM, Matt Burgess <mattyb...@apache.org> wrote:
> There are incremental data movement processors in NiFi depending on
> your source/target. For example, if your sources are files, you can
> use ListFile in combination with FetchFile, the former will keep track
> of which files it has found thus far, so if you put new files into the
> location (or update existing ones), only those new/updated files will
> be processed the next time.
> For database (RDBMS) sources, there are the QueryDatabaseTable and
> GenerateTableFetch processors, which support the idea of "maximum
> value columns", such that for each of said columns, the processor(s)
> will keep track of the maximum value observed in that column, then for
> future executions of the processor, only rows whose values in those
> columns exceed the currently-observed maximum will be retrieved, then
> the maximum will be updated, and so forth.
> The Usage documentation for these processors can be found at
> https://nifi.apache.org/docs.html (left-hand side under Processors).
> On Thu, Oct 13, 2016 at 11:05 AM, Gop Krr <gop....@gmail.com> wrote:
> > Hi All,
> > I am learning NiFi as well as trying to deploy it in production for few
> > cases. One of the use case is ETL and another use case is, using NiFi as
> > backup solution, where it takes the data from one source and moves to
> > another database|file. Is anyone using NiFi for this purpose? Does NiFi
> > support incremental data move?
> > It would be awesome if someone can point me to right documentation.
> > Thanks
> > Rai