[
https://issues.apache.org/jira/browse/HIVE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19340:
---------------------------------------
Description: The transactions opened by applying EVENT_OPEN_TXN should
never be aborted automatically due to time-out. Aborting of transaction started
by replication task may leads to inconsistent state at target which needs
additional overhead to clean-up. So, it is proposed to mark the transactions
opened by replication task as special ones and shouldn't be aborted if heart
beat is lost. This helps to ensure all ABORT and COMMIT events will always find
the corresponding txn at target to operate. (was:
h1. Replicate ACID write Events
* Create new EVENT_WRITE event with related message format to log the write
operations with in a txn along with data associated.
* Log this event when perform any writes (insert into, insert overwrite, load
table, delete, update, merge, truncate) on table/partition.
* If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple
partitions, then need to log one event per partition.
* DbNotificationListener should log this type of event to special metastore
table named "MTxnWriteNotificationLog".
* This table should maintain a map of txn ID against list of tables/partitions
written by given txn.
* The entry for a given txn should be removed by the cleaner thread that
removes the expired events from EventNotificationTable.
h1. Replicate Commit Txn operation (with writes)
Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions
modified within the txn.
*Source warehouse:*
* This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog"
metastore table to consolidate the list of tables/partitions modified within
this txn scope.
* Based on the list of tables/partitions modified and table Write ID, need to
compute the list of delta files added by this txn.
* Repl dump should read this message and dump the metadata and delta files
list.
*Target warehouse:*
* Ensure snapshot isolation at target for on-going read txns which shouldn't
view the data replicated from committed txn. (Ensured with open and allocate
write ID events).)
> Disable timeout of transactions opened by replication task at target cluster
> ----------------------------------------------------------------------------
>
> Key: HIVE-19340
> URL: https://issues.apache.org/jira/browse/HIVE-19340
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
> Affects Versions: 3.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> The transactions opened by applying EVENT_OPEN_TXN should never be aborted
> automatically due to time-out. Aborting of transaction started by replication
> task may leads to inconsistent state at target which needs additional
> overhead to clean-up. So, it is proposed to mark the transactions opened by
> replication task as special ones and shouldn't be aborted if heart beat is
> lost. This helps to ensure all ABORT and COMMIT events will always find the
> corresponding txn at target to operate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
