Hi Timothy ,

Hope you are doing well. We have been using your data flow(
https://community.hortonworks.com/content/kbentry/155527/ingesting-golden-gate-records-from-apache-kafka-an.html#
)
with slight modifications to store the data in Hbase. To version the rows
we have been using Op_ts of golden gate json . But now we have found that
multiple transactions can have the same Op_ts.  e.g. both update or delete
can have the same Op_ts and if they arrive out of order to the PutHbaseJson
processor then it can cause the target table to go out of sync. I am using
the a cluster of nifi nodes so i cannot use Enforceorder processor to order
the kafka messages as i understand it only order the flow files on a single
node only and not across the cluster. Additionally we have a separate topic
for each table and we have several consumer groups. I tried using the
Current_ts column of the golden gate message but then if GG abends and
restart the replication it will send the past data with the newer
current_ts which will also cause the un-sync. I was wondering if you can
give any idea so that we can order our transaction correctly.

Regards,
Faisal

Reply via email to