Hello Marton Greber, Alexey Serbin, Zoltan Martonka, Ashwani Raina, Kudu
Jenkins, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24578
to look at the new patch set (#10).
Change subject: KUDU-3800 Add diff-scan option to return unobservable rows
......................................................................
KUDU-3800 Add diff-scan option to return unobservable rows
A diff scan reports rows whose state changed inside the
(snap_start_timestamp, snap_timestamp] range. Rows live at
snap_start_timestamp and deleted before snap_timestamp are surfaced
via 'include_deleted_rows'; rows inserted or updated inside the range
that are still live at snap_timestamp are surfaced by default. But rows
whose entire lifecycle (INSERT -> eventual DELETE) is contained inside
the range exist at neither endpoint, so no snapshot read would see
them, and 'include_deleted_rows' does not apply. It only surfaces
rows that were live at snap_start_timestamp. The diff scan drops them
even though a two-tier system that uses diff scans to keep a second
tier consistent with Kudu would want them: a row previously migrated
out of Kudu (and whose delete was propagated) that is re-inserted and
re-deleted inside a diff-scan window is not reported today, so any
stale copy downstream is never reconciled.
Note: This is NOT change-data-capture. The diff scan collapses all
mutations for a row within the range into a single output row
reflecting its state at snap_timestamp; intermediate UPDATE values and
mutation counts are not preserved. Callers needing per-mutation
history need a different mechanism.
This patch adds an optional 'include_unobservable_rows' knob to the
NewScanRequestPB and plumb it through:
- ScanConfiguration / KuduScanner. A new SetDiffScan overload takes a
'DiffScanRowVisibility' enum (OBSERVABLE_ONLY / INCLUDE_UNOBSERVABLE)
rather than a raw bool, so the intent is legible at call sites.
- ScanTokenPB and KuduScanTokenBuilder, so token-based clients can carry
the flag over the wire.
- RowIteratorOptions -> MemRowSet iterator and DeltaApplier /
SelectedDeltas, where the "drop unobservable" test now short-circuits
when the caller opts in. Such rows are returned marked deleted via the
IS_DELETED virtual column.
The tablet server enforces the invariant uniformly: it returns
INVALID_SCAN_SPEC if the flag is set without snap_start_timestamp
(regardless of read mode), and on diff scans whose projection lacks
an IS_DELETED virtual column. DCHECKs in DeltaApplier and MemRowSet
catch in-process callers that build a RowIteratorOptions with the flag
set outside a valid diff-scan configuration.
The RPC field is a new 'optional enum' and is wire-compatible with
older servers (which ignore it) and older clients (which don't set it).
Corresponding tests have been added in tablet_server-test.cc, client-test.cc
and diff_scan-test.cc.
The three-argument SetDiffScan() overloads on KuduScanner and
KuduScanTokenBuilder do not have KUDU_NO_EXPORT unlike their older
counterparts as the intention is to make this user accessible.
Followup work to include Java and Python client support.
Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
---
M src/kudu/client/client-test.cc
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/client.proto
M src/kudu/client/scan_configuration.cc
M src/kudu/client/scan_configuration.h
M src/kudu/client/scan_token-internal.cc
M src/kudu/client/scanner-internal.cc
M src/kudu/common/common.proto
M src/kudu/tablet/delta_applier.cc
M src/kudu/tablet/delta_store.cc
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/memrowset.cc
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/tablet.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver.proto
20 files changed, 752 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/24578/10
--
To view, visit http://gerrit.cloudera.org:8080/24578
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
Gerrit-Change-Number: 24578
Gerrit-PatchSet: 10
Gerrit-Owner: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
