[
https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-19089:
---------------------------------------
Description:
*EVENT_ALLOCATE_WRITE_ID*
*Source Warehouse:*
* Create new event type EVENT_ALLOCATE_WRITE_ID with related message format
etc.
* Capture this event when allocate a table write ID from the sequence table by
ACID operation.
* Repl dump should read this event from EventNotificationTable and dump the
message.
*Target Warehouse:*
* Repl load should read the event from the dump and get the message.
* Validate if source txn ID from the event is there in the source-target txn
ID map. If not there, just noop the event.
* If valid, then Allocate table write ID from sequence table
*Extend listener notify event API to add two new parameter , dbconn and
sqlgenerator to add the events to notification_log table within the same
transaction*
was:
*EVENT_OPEN_TXN:*
*Source Warehouse:*
- Create new event type EVENT_OPEN_TXN with related message format etc.
- When any transaction is opened either by auto-commit mode or multi-statement
mode, need to capture this event.
- Repl dump should read this event from EventNotificationTable and dump the
message.
*Target Warehouse:*
- Repl load should read the event from the dump and get the message.
- Open a txn in target warehouse.
- Create a map of source txn ID against target txn ID and persist the same in
metastore. There should be one map per replication policy (DBName.* incase of
DB level replication, DBName.TableName incase of table level replication)
*EVENT_COMMIT_TXN (Without writes)*
Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions
modified within the txn.
*Source warehouse:*
- Create EVENT_COMMIT_TXN event type with corresponding message format etc.
*Target warehouse:*
- Repl load should read this event from the dump.
- Validate the source txn ID from the event using the Source-Target Txn ID map
maintained in target metastore. Also, need to check if corresponding target txn
ID is valid.
- If valid, then apply the event and commit the corresponding target
transaction.
- This new event should be idempotent such that if it is applied twice, then
second time it should be loop.
*EVENT_ABORT_TXN*
Source Warehouse:
- Create new event type EVENT_ABORT_TXN with related message format etc.
- Capture this event when abort the txn.
- Repl dump should read this event from EventNotificationTable and dump the
message.
*Target Warehouse:*
- Repl load should read the event from the dump and get the message.
- Validate if source txn ID from the event is there in the source-target txn
ID map. If not there, just noop the event.
- If valid, then Abort the corresponding target txn and remove the entry from
source-target txn ID map.
All these new events should be idempotent such that if it is applied twice,
then second time it should be noop.
> Create/Replicate Allocate write-id event
> ----------------------------------------
>
> Key: HIVE-19089
> URL: https://issues.apache.org/jira/browse/HIVE-19089
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
> Affects Versions: 3.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: ACID, DR, replication
> Fix For: 3.0.0
>
>
> *EVENT_ALLOCATE_WRITE_ID*
> *Source Warehouse:*
> * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format
> etc.
> * Capture this event when allocate a table write ID from the sequence table
> by ACID operation.
> * Repl dump should read this event from EventNotificationTable and dump the
> message.
> *Target Warehouse:*
> * Repl load should read the event from the dump and get the message.
> * Validate if source txn ID from the event is there in the source-target txn
> ID map. If not there, just noop the event.
> * If valid, then Allocate table write ID from sequence table
> *Extend listener notify event API to add two new parameter , dbconn and
> sqlgenerator to add the events to notification_log table within the same
> transaction*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
