Hi Andrew, Thank you for your suggestion. We are using the timestamp property of the PutHbase processor to enforce the order. This timestamp is extracted from the golden gate message as a meta data. I agree with your approach for creating an End of day file which seems to be the most logical way of doing the reconciliation.
Thank you for the help. Faisal On Mon, Jun 11, 2018 at 6:07 PM Andrew Psaltis <psaltis.and...@gmail.com> wrote: > Hi Faisal, > OK, so then a single partition in Kafka and I assume that you are using > the EnforceOrder processor or another means to ensure the records are in > order so that updates happen after inserts and so forth. In that case, > assuming everything is in order the simplest approach is just to do a count > at the end-of-the-day, whatever that means for your business. However, > often times a simple count is not good enough, as you actually want to know > that the tables reconcile. Often times, in this case, there is a way to > produce an End Of Day file or record for the source system. This will > usually contain things like the record count, along with a sum of various > columns, and possibly a query that is used to produce the sum or another > validation. With this, you can then ensure that the tables do reconcile. > > Do you have the ability to create an "End of Day" file or something along > those lines that you can use? > > > [1] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.EnforceOrder/index.html > > Thanks, > Andrew > > > > On Mon, Jun 11, 2018 at 11:11 AM Faisal Durrani <te04.0...@gmail.com> > wrote: > >> Hi Andrew, >> We are receiving the golden gate transactions from Kafka which is >> received in Nifi through consume kafka processor . Our data flow then >> reduces the golden gate json message and sends the data to the target table >> in Hbase using the PutHbase Json processor. >> >> Thanks, >> Faisal >> >> On Mon, Jun 11, 2018 at 12:01 PM Andrew Psaltis <psaltis.and...@gmail.com> >> wrote: >> >>> Hi Faisal, >>> There are various ways this can be handled. But this is going to depend >>> on, how are you receiving data from Oracle via Golden Gate. Are you using >>> the HBase Handler, the HDFS Handler, a Flat File, Kafka, or via another >>> means? >>> >>> Thanks, >>> Andrew >>> >>> On Mon, Jun 11, 2018 at 9:43 AM Faisal Durrani <te04.0...@gmail.com> >>> wrote: >>> >>>> Is there a recommended way to ensure the row counts form tables in >>>> source (Oracle) are consistent with that of target tables in Hbase ( >>>> data-lake)? .We are using Nifi which receives the golden gate messages and >>>> then by using different processor we store the transactions in Hbase ,so >>>> essentially the tables in Hbase should be in sync with the tables in Oracle >>>> at all times. I am interested in knowing how the teams ensure and proof >>>> this ? Do they take row counts from source and target everyday and match it >>>> and say that its synced ? I used the counter option in Nifi which >>>> maintained the record received against each table but i guess that is not >>>> an optimized way to do it. >>>> >>>
