Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16515 )
Change subject: KUDU-2612: add txn memrowsets to tablet ...................................................................... KUDU-2612: add txn memrowsets to tablet This patch introduces the ability to insert data into per-transaction MRSs. When a transaction is started, it creates a new MRS to which inserts of that transaction can be applied. This "uncommitted transactional MRS" cannot be inserted to by any other transaction and cannot be scanned by any scanner. Once the transaction is committed, i.e. the FINALIZE_COMMIT op is applied for the transaction, the MRS is moved to a separate set of "committed" transactional MRSs that are available to be scanned, updated, and flushed. If the transaction is aborted, the transactional MRS is excised from the list of transactional MRSs entirely. When inserting, updating, or scanning rows, ops must now consider all committed transactional MRSs, in addition to the main, non-transactional MRS, checking for row presence in any store that is "committed" and visible to users. Additionally, when a MRS flush is performed, rather than flushing the main, non-transactional MRS alone, Kudu will now collect the non-transactional MRS along with all committed transactional MRSs, and roll DRSs using the same rowset-merging code that exists for compactions. The new flushed data will honor each transactions' commit timestamp rather than the per-row apply timestamps. Existing MRS-related metrics have been updated to reflect that such flush ops now need to consider transactional MRSs. In order to support rebuilding these MRSs when restarting, a new boolean field is added to the transaction metadata: 'flushed_committed_mrs', which gets set when the transactional MRS is flushed to disk. If true, the transaction's write ops do not need to be replayed, since the transaction does not have an active MRS. Otherwise, the transaction's write ops do need to be replayed, even if the transaction is persisted in metadata as being committed. Additionally, a transaction ID is added to the MutatedStorePB message stored in the WAL, for the sake of replaying ops that land in committed, transactional MRSs. Upon replay of such mutations, if the transactional MRS being mutated to is not active (i.e. it has been flushed), the mutation op can be ignored. When starting up, we start transactional MRSs for all transactions that do not have 'flushed_committed_mrs' set, which indicates that the transaction's MRS was not flushed before restarting. NOTE: the locking story here is not complete -- multiple transactions can write to the same rows without consequence, which violates our row uniqueness constraints. This will be addressed in future patches. Change-Id: I1dea4031c0acb875993352d452dc4c233e35a502 Reviewed-on: http://gerrit.cloudera.org:8080/16515 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin <[email protected]> --- M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/txn_participant-itest.cc M src/kudu/tablet/memrowset.cc M src/kudu/tablet/memrowset.h M src/kudu/tablet/metadata.proto M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/row_op.cc M src/kudu/tablet/row_op.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet.proto M src/kudu/tablet/tablet_bootstrap.cc M src/kudu/tablet/tablet_metadata.cc M src/kudu/tablet/tablet_metadata.h M src/kudu/tablet/tablet_replica-test-base.cc M src/kudu/tablet/txn_metadata.h M src/kudu/tablet/txn_participant-test.cc M src/kudu/tablet/txn_participant.cc M src/kudu/tablet/txn_participant.h M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tserver.proto M src/kudu/util/locks.h 23 files changed, 1,368 insertions(+), 238 deletions(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16515 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1dea4031c0acb875993352d452dc4c233e35a502 Gerrit-Change-Number: 16515 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Wong <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
