Re: Ingesting golden gate messages to Hbase using Nifi

glad it is working for you, Faisal. I am sure you already know that current_ts timestamp value will change every time you reprocess GG trail files in case of GG server failure, checkpoint rollbacks or when you reposition GG with a command like ALTER REPLICAT xxx, EXTSEQNO 5, EXTRBA 0 or ALTER

Re: Ingesting golden gate messages to Hbase using Nifi

Hi Timothy, Thank you for your input. Hi Boris, Hope you are doing well, I wanted to update you with solution which finally worked for me. I found out the the current_ts column of the Golden gate message was unique for all the transactions and in addition to that it also has millisecond

Re: Ingesting golden gate messages to Hbase using Nifi

Enforcing order is really tricky with Kafka. The only way to enforce order is to reduce the # of nodes processing. You can have one NiFi master node read from Kafka and have it distribute the workload to other NiFi nodes and force ordering. Or you may want to batch them up into say 10-15

Re: Ingesting golden gate messages to Hbase using Nifi

Faisal, BTW I stumbled upon this doc, that explains how HBase GoldenGate handler works in a similar scenario you've described: https://docs.oracle.com/goldengate/bd123210/gg-bd/GADBD/using-hbase-handler.htm#GADBD-GUID-1A9BA580-628B-48BD-9DC0-C3DF9722E0FB They provide an option to generate

Re: Ingesting golden gate messages to Hbase using Nifi

Sorry I meant RBA.GG has a bunch of tokens you can add to your json file - you can even create your own. POS should be good and if op_ts does not work for you, why not to generate your own timestamp using POS? (Now() expression). You also add another token that identifies

Re: Ingesting golden gate messages to Hbase using Nifi

Hi, Regarding: "So I'm not sure how do i combine all the flow files together on a single node? ( I know how to distribute them i.e is by using S2S-RPG) " I wanted to mention that it's now possible, with NiFi 1.8.0, to send back flow files on a single node.

Re: Ingesting golden gate messages to Hbase using Nifi

Hi Boris, Thank you for your reply. Let me try explaining my data flow in detail. I am receiving the GG transaction as JSON format through Kafka so I can only use the fields provided by the Kafka handler of GG ( Json plug-gable format). I think you meant RBA value instead of rbc. I don't think

Re: Ingesting golden gate messages to Hbase using Nifi

Hi Faisal, I am not Timothy, but you raise an interesting problem we might face soon as well. I did not expect the situation you described and I thought transaction time would be different. Our intent was to use op_ts to enforce order but another option is to use GG rbc value or oracle rowscn

Ingesting golden gate messages to Hbase using Nifi

Hi Timothy , Hope you are doing well. We have been using your data flow( https://community.hortonworks.com/content/kbentry/155527/ingesting-golden-gate-records-from-apache-kafka-an.html# ) with slight modifications to store the data in Hbase. To version the rows we have been using Op_ts of golden