[
https://issues.apache.org/jira/browse/PHOENIX-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356256#comment-15356256
]
Poorna Chandra commented on PHOENIX-2993:
-----------------------------------------
bq. What about having the transaction manager track the table names involved in
the commit?
Invalid list is used for exceptional cases when the transaction manager cannot
determine the state of a transaction. This typically happens when a transaction
client crashes after starting a transaction.
A transaction client performs the data operations on HBase directly without
involving the transaction manager. A client can crash before it updates the
transaction manager with the state of the transaction. In such cases the
transaction manager has no details on the data operations the client made
whatsoever.
In the case the client does not crash on a failure, it will revert the data
operations, and abort the transaction. In this case the transaction does not
become invalid.
So in either of the cases the transaction manager doesn't need to track the
tables involved in a transaction.
bq. Basically, we'd need to check every single HBase table and wait until all
of them have been major compacted which seems kind of painful when 90% of the
time there's a single table involved.
Since a transaction could have been used to update any table, we have to make
sure all tables are major compacted before we can proceed. However, we can
mitigate the wait by doing the following -
# Record the major compaction status for regions at regular interval, and try
to find the best fitting minimum compaction time across all active regions at a
given time
# Cache invalid list on region servers - this will allow Tephra to support a
few hundred thousand invalid ids without performance degradation. This allows
the system to tolerate a lower frequency of major compactions.
# Record min/max transaction ids of a region during major compaction - this
will allow us to ignore regions that are slow to compact. In that case we
cannot remove invalid ids between min and max for that region.
We are implementing point 1 in the first cut. Points 2 and 3 will be added
later.
> Tephra: Prune invalid transaction set once all data for a given invalid
> transaction has been dropped
> ----------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-2993
> URL: https://issues.apache.org/jira/browse/PHOENIX-2993
> Project: Phoenix
> Issue Type: New Feature
> Reporter: Poorna Chandra
> Assignee: Poorna Chandra
> Attachments: ApacheTephraAutomaticInvalidListPruning.pdf
>
>
> From TEPHRA-35 -
> In addition to dropping the data from invalid transactions we need to be able
> to prune the invalid set of any transactions where data cleanup has been
> completely performed. Without this, the invalid set will grow indefinitely
> and become a greater and greater cost to in-progress transactions over time.
> To do this correctly, the TransactionDataJanitor coprocessor will need to
> maintain some bookkeeping for the transaction data that it removes, so that
> the transaction manager can reason about when all of a given transaction's
> data has been removed. Only at this point can the transaction manager safely
> drop the transaction ID from the invalid set.
> One approach would be for the TransactionDataJanitor to update a table
> marking when a major compaction was performed on a region and what
> transaction IDs were filtered out. Once all regions in a table containing the
> transaction data have been compacted, we can remove the filtered out
> transaction IDs from the invalid set. However, this will need to cope with
> changing region names due to splits, etc.
> Note: This will be moved to Tephra JIRA once the setup of Tephra JIRA is
> complete (INFRA-11445)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
