As an interesting architectural approach we took eons ago, before NiFi, was to take daily snapshots of a full table. Every row would then be hashed/digested or in any other way uniquely identified and 2 datasets would be crossed and compared to find inserts/deletes/updates. It was involved, but worked.
Andrew On Sat, Sep 16, 2017, 2:38 AM Uwe Geercken <uwe.geerc...@web.de> wrote: > Bryan, > > yes, the change log would be possible. In my use case I have Oracle 11 as > the source - and I can not change the source easily (takes long - is > expensive). > > I was expecting this answer but wanted to make sure that I have not missed > anything. I will try to build my use case around something else then. > > Thanks for your response(s). > > Rgds, > > Uwe > > *Gesendet:* Freitag, 15. September 2017 um 16:15 Uhr > *Von:* "Bryan Bende" <bbe...@gmail.com> > *An:* users@nifi.apache.org > *Betreff:* Re: QueryDatabaseTable - Deleted Records > Uwe, > > Typically you need to process the change log of the database in this > case, which unfortunately usually becomes database specific. > > I believe we have a processor CaptureChangeMySQL that can process the > MySQL change log. > > -Bryan > > > On Tue, Sep 12, 2017 at 1:39 PM, Uwe Geercken <uwe.geerc...@web.de> wrote: > > Hello, > > > > apparently the QueryDatabaseTable processor catches changes made to the > data > > of the source database - updates and inserts. > > > > Has anybody a good idea or strategy how to handle deletes in the source > > database? Of course one could flag a record as deleted instead of > phisically > > deleting it. But this means changing the source system in many cases and > > that is sometimes not possible. And yes, if you process the change log > (if > > available) of the source system that is also a good option. > > > > Would be greatful for any tips or a best practive of how you do it. > > > > Rgds, > > > > Uwe >