Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24578 )
Change subject: KUDU-3800 Add diff-scan option to return unobservable rows ...................................................................... KUDU-3800 Add diff-scan option to return unobservable rows A diff scan reports rows whose state changed inside the (snap_start_timestamp, snap_timestamp] range. Rows live at snap_start_timestamp and deleted before snap_timestamp are surfaced via 'include_deleted_rows'; rows inserted or updated inside the range that are still live at snap_timestamp are surfaced by default. But rows whose entire lifecycle (INSERT -> eventual DELETE) is contained inside the range exist at neither endpoint, so no snapshot read would see them, and 'include_deleted_rows' does not apply. It only surfaces rows that were live at snap_start_timestamp. The diff scan drops them even though a two-tier system that uses diff scans to keep a second tier consistent with Kudu would want them: a row previously migrated out of Kudu that is re-inserted and re-deleted inside a diff-scan window is not reported today, so any stale copy downstream is never reconciled. Note: This is NOT change-data-capture. The diff scan collapses all mutations for a row within the range into a single output row reflecting its state at snap_timestamp; intermediate UPDATE values and mutation counts are not preserved. Callers needing per-mutation history need a different mechanism. This patch adds an optional 'row_visibility' knob to the NewScanRequestPB and plumbs it through: - ScanConfiguration / KuduScanner. A new SetDiffScan overload takes a 'DiffScanRowVisibility' enum (OBSERVABLE_ONLY / INCLUDE_UNOBSERVABLE) so the intent is legible at call sites. - ScanTokenPB and KuduScanTokenBuilder, so token-based clients can carry the flag over the wire. - RowIteratorOptions -> MemRowSet iterator and DeltaApplier / SelectedDeltas, where the "drop unobservable" test now short-circuits when the caller opts in. Such rows are returned marked deleted via the IS_DELETED virtual column. The tablet server enforces the invariant uniformly: it returns INVALID_SCAN_SPEC if the flag is set without snap_start_timestamp (regardless of read mode), and on diff scans whose projection lacks an IS_DELETED virtual column. DCHECKs in DeltaApplier and MemRowSet catch in-process callers that build a RowIteratorOptions with the flag set outside a valid diff-scan configuration. The RPC field is a new 'optional enum' and is wire-compatible with older servers (which ignore it) and older clients (which don't set it). Corresponding tests have been added in tablet_server-test.cc, client-test.cc, scan_token-test.cc and diff_scan-test.cc. The prior two argument SetDiffScan() (which was never part of the client ABI) is now replaced with three argument SetDiffScan() and exported for user code but documented private. Followup work to include Java client support. Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a Reviewed-on: http://gerrit.cloudera.org:8080/24578 Reviewed-by: Ashwani Raina <[email protected]> Tested-by: Abhishek Chennaka <[email protected]> --- M src/kudu/client/client-test.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/client.proto M src/kudu/client/scan_configuration.cc M src/kudu/client/scan_configuration.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc M src/kudu/client/scanner-internal.cc M src/kudu/common/common.proto M src/kudu/integration-tests/fuzz-itest.cc M src/kudu/tablet/delta_applier.cc M src/kudu/tablet/delta_store.cc M src/kudu/tablet/delta_store.h M src/kudu/tablet/diff_scan-test.cc M src/kudu/tablet/memrowset.cc M src/kudu/tablet/rowset.cc M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.h M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tserver.proto 22 files changed, 941 insertions(+), 31 deletions(-) Approvals: Ashwani Raina: Looks good to me, approved Abhishek Chennaka: Verified -- To view, visit http://gerrit.cloudera.org:8080/24578 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a Gerrit-Change-Number: 24578 Gerrit-PatchSet: 20 Gerrit-Owner: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Martonka <[email protected]>
