[
https://issues.apache.org/jira/browse/HIVE-26265?focusedWorklogId=784029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-784029
]
ASF GitHub Bot logged work on HIVE-26265:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Jun/22 04:10
Start Date: 23/Jun/22 04:10
Worklog Time Spent: 10m
Work Description: cmunkey commented on PR #3365:
URL: https://github.com/apache/hive/pull/3365#issuecomment-1163905485
Changed the following since the last upload:
- Eliminated HMS client changes. The HMS client changes were used to query
HMS to get the databases updated for a AbortTxnEvent, in the case where the
event was used to notify the non transactional listeners. Currently only the
DbTxnListener which is a transactional listener needs the databases updated.
- Added unit test to make sure that old versions of the AbortTxnEvent can
still be serialiazed for the case where REPL DUMP may be processing events
logged by an older version.
- Modified TestReplicationFilters to do extra transactions on the replica
database to test out the source to target
transaction mapping. Currently this test is causing the following problems
that I am still debugging:
testTxnEventsOptimized() - to assert that transaction mapping was being
done correctly, I tried to check the REPL_TXN_MAP on the replica. But this
proved difficult because of the following:
o REPL_TXN_MAP is cleared during commit/abort, so it could not be
checked at the end of the test. Instead, I tried to capture the table in the
openTxn test listener. But it seemed like the extra SELECT that the test is
doing is causing some kind of deadlock in the code. Some of the sampling did
work, and I could see in the debugger the mapping, after a few times sampling,
the test gets a "org.apache.hadoop.hive.ql.lockmgr.LockException: Error
communicating with the metastore" exception.
o For the test to pass, I commented out the sampling of the REPL_TXN_MAP
table.
testTxnEventsUnoptimized() - this these was modified to burn some
transactions on the replica also. But when these extra transactions are burned,
on the replica, there is now extra openTxn() calls that are unexpected. This
test has the transaction filtering as OFF, so it should be default behavior, so
I am not sure what is going on. For the test to pass, I commented out the code
to burn the transactions.
Issue Time Tracking
-------------------
Worklog Id: (was: 784029)
Time Spent: 1h 20m (was: 1h 10m)
> REPL DUMP should filter out OpenXacts and unneeded CommitXact/Abort.
> --------------------------------------------------------------------
>
> Key: HIVE-26265
> URL: https://issues.apache.org/jira/browse/HIVE-26265
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Reporter: francis pang
> Assignee: francis pang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> REPL DUMP is replication all OpenXacts, even when they are from other non
> replicated databases. This wastes space in the dump, and ends up opening
> unneeded transactions during REPL LOAD.
>
> Add a config property for replication that filters out OpenXact events during
> REPL DUMP. During REPL LOAD, the txns can be implicitly opened when the
> ALLOC_WRITE_ID is processed. For CommitTxn and AbortTxn, dump only if WRITE
> ID was allocated.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
