Manikandan, QueryDatabaseTable (and GenerateTableFetch) should work for incremental fetching of rows, provided that you have at least one column that strictly increases (like a primary key with integer ID for example). In that case you put the column name into the processor's Maximum Value Columns property. The first time the processor runs, it will get all rows from the table, but then it will keep track of the largest value it has seen for the maximum value column. The next time it runs, it will only retrieve rows whose values in that column are greater than the currently observed maximum.
For your example, did you select a maximum value column and include it in the processor's Maximum Value Columns property? Regards, Matt On Thu, Sep 29, 2016 at 6:15 AM, Manik <[email protected]> wrote: > Hi Team, > > I am confused to use which processor is fit for incremental fetch in nifi. > In nifi My task is like i need to fetch the data from sql server and doing > Incremental fetch data and load into hadoop and i need to compare two datas > i.e(incremantal and normal datas) merged and load into hive .In this task > i already used some processor it didnt work properly .please give some about > this task .I already use this processor > > QueryDatabasetable--->>--SplitAvro-->>--ConvertAvroToJSON-->>--ReplaceText-->>--LogAttributes-->>--PutHdfs-->>--MergeContent-->>UpdateAttribute-->>--PutHiveStreaming > > In this processor I didnt get exact Incremental data as well as Base data > .please guide me to solve this tasks.. > > > Thanks&Advance, > > Manikandan > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
