[
https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410960#comment-16410960
]
mahesh kumar behera commented on HIVE-18781:
--------------------------------------------
[~ekoifman] and [~sankarh] please review the patch
Changes Done
# Added Events for open/commit/abort transaction
# Added handling for these events during replication load
# For adding the events to notification_log table, direct SQL execution is
done instead of through DN. This is to make sure the event logging and the
actual operation is done within a transaction.
# At DbNotificationListener, handling for open/commit/abort is done
differently to make sure they use the same connection object to do the SQL
execution.
# These events constructor is modified to store the connection object.
> Create/Replicate Open, Commit (without writes) and Abort Txn events
> -------------------------------------------------------------------
>
> Key: HIVE-18781
> URL: https://issues.apache.org/jira/browse/HIVE-18781
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
> Affects Versions: 3.0.0
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch,
> HIVE-18781.03.patch, HIVE-18781.04.patch, HIVE-18781.05.patch,
> HIVE-18781.06.patch, HIVE-18781.07.patch, HIVE-18781.08.patch,
> HIVE-18781.08.patch
>
>
> *EVENT_OPEN_TXN:*
> *Source Warehouse:*
> - Create new event type EVENT_OPEN_TXN with related message format etc.
> - When any transaction is opened either by auto-commit mode or
> multi-statement mode, need to capture this event.
> - Repl dump should read this event from EventNotificationTable and dump the
> message.
> *Target Warehouse:*
> - Repl load should read the event from the dump and get the message.
> - Open a txn in target warehouse.
> - Create a map of source txn ID against target txn ID and persist the same
> in metastore. There should be one map per replication policy (DBName.* incase
> of DB level replication, DBName.TableName incase of table level replication)
>
> *EVENT_COMMIT_TXN (Without writes)*
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions
> modified within the txn.
> *Source warehouse:*
> - Create EVENT_COMMIT_TXN event type with corresponding message format etc.
> *Target warehouse:*
> - Repl load should read this event from the dump.
> - Validate the source txn ID from the event using the Source-Target Txn ID
> map maintained in target metastore. Also, need to check if corresponding
> target txn ID is valid.
> - If valid, then apply the event and commit the corresponding target
> transaction.
> - This new event should be idempotent such that if it is applied twice, then
> second time it should be loop.
>
> *EVENT_ABORT_TXN*
> Source Warehouse:
> - Create new event type EVENT_ABORT_TXN with related message format etc.
> - Capture this event when abort the txn.
> - Repl dump should read this event from EventNotificationTable and dump the
> message.
> *Target Warehouse:*
> - Repl load should read the event from the dump and get the message.
> - Validate if source txn ID from the event is there in the source-target txn
> ID map. If not there, just noop the event.
> - If valid, then Abort the corresponding target txn and remove the entry
> from source-target txn ID map.
> All these new events should be idempotent such that if it is applied twice,
> then second time it should be noop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
