[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8962 ) Change subject: consensus: Reduce copy/paste for observer callbacks .. consensus: Reduce copy/paste for observer callbacks This is a net-negative line patch that unifies how the PeerMessageQueue notifies PeerMessageQueueObserver instances that are registered with it. The patch removes most of the copy/paste in this part of the code. This patch has no functional changes. Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Reviewed-on: http://gerrit.cloudera.org:8080/8962 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/consensus_queue.h 2 files changed, 35 insertions(+), 83 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8962 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Gerrit-Change-Number: 8962 Gerrit-PatchSet: 3 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] mini cluster: Remove tablet server index by uuid() from interface
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8963 ) Change subject: mini_cluster: Remove tablet_server_index_by_uuid() from interface .. mini_cluster: Remove tablet_server_index_by_uuid() from interface The tablet_server_index_by_uuid() method, which common to the InternalMiniCluster and ExternalMiniCluster, isn't really useful from the parent interface (MiniCluster) and isn't compatible with a MixedMiniCluster that I'm working on. Let's just remove it from the pure virtual parent interface. Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab Reviewed-on: http://gerrit.cloudera.org:8080/8963 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.h 3 files changed, 6 insertions(+), 6 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab Gerrit-Change-Number: 8963 Gerrit-PatchSet: 3 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8962 ) Change subject: consensus: Reduce copy/paste for observer callbacks .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8962 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Gerrit-Change-Number: 8962 Gerrit-PatchSet: 2 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 09 Jan 2018 02:43:14 + Gerrit-HasComments: No
[kudu-CR] mini cluster: Remove tablet server index by uuid() from interface
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8963 ) Change subject: mini_cluster: Remove tablet_server_index_by_uuid() from interface .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab Gerrit-Change-Number: 8963 Gerrit-PatchSet: 2 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Tue, 09 Jan 2018 02:43:03 + Gerrit-HasComments: No
[kudu-CR] [tablet] Add a pre-compaction check to avoid compactions in the past
Hello Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8887 to look at the new patch set (#6). Change subject: [tablet] Add a pre-compaction check to avoid compactions in the past .. [tablet] Add a pre-compaction check to avoid compactions in the past This adds a pre-compaction check that gets the max timestamp of all the rows that will be considered for compaction and makes sure that timestamp "may" be committed under the compaction snapshot. That is that the max timestamp among the rows is not so far in the future, relative to the compaction snapshot, that that the compaction would rewind committed row state. This is generalization of the repro introduced in previous patches for KUDU-2233 but that now applies to all compactions in the past and not only those which have duplicated rows in the inputs. This is not a fix yet, that's left for follow up patches. Change-Id: I80edc9dc8d5eec475998a9c721b9f591eb6180dc --- M src/kudu/tablet/compaction.cc M src/kudu/tablet/compaction.h M src/kudu/tablet/delta_tracker.cc M src/kudu/tablet/delta_tracker.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset.cc M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/rowset.cc M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.cc 12 files changed, 153 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/87/8887/6 -- To view, visit http://gerrit.cloudera.org:8080/8887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I80edc9dc8d5eec475998a9c721b9f591eb6180dc Gerrit-Change-Number: 8887 Gerrit-PatchSet: 6 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8962 to look at the new patch set (#2). Change subject: consensus: Reduce copy/paste for observer callbacks .. consensus: Reduce copy/paste for observer callbacks This is a net-negative line patch that unifies how the PeerMessageQueue notifies PeerMessageQueueObserver instances that are registered with it. The patch removes most of the copy/paste in this part of the code. This patch has no functional changes. Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 --- M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/consensus_queue.h 2 files changed, 35 insertions(+), 83 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/8962/2 -- To view, visit http://gerrit.cloudera.org:8080/8962 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Gerrit-Change-Number: 8962 Gerrit-PatchSet: 2 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Mike Percy has posted comments on this change. (
http://gerrit.cloudera.org:8080/8962 )
Change subject: consensus: Reduce copy/paste for observer callbacks
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8962/1/src/kudu/consensus/consensus_queue.cc
File src/kudu/consensus/consensus_queue.cc:
http://gerrit.cloudera.org:8080/#/c/8962/1/src/kudu/consensus/consensus_queue.cc@1323
PS1, Line 1323: void
PeerMessageQueue::NotifyObserversTask(std::function
func) {
> seems accurate
Done
--
To view, visit http://gerrit.cloudera.org:8080/8962
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437
Gerrit-Change-Number: 8962
Gerrit-PatchSet: 1
Gerrit-Owner: Mike Percy
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 09 Jan 2018 01:47:40 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2233 Add a test case for compactions in the past
Hello Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8885 to look at the new patch set (#6). Change subject: KUDU-2233 Add a test case for compactions in the past .. KUDU-2233 Add a test case for compactions in the past In some cases we're performing compactions before the clock has been updated at all, making the compaction snapshot an empty one. This makes it so the compaction runs on a snapshot that discards all redo deltas and applies all undos. When a compaction with such a snapshot finds two duplicate rows (a ghost and live row) it can't distinguish the two because the redo DELETE of the ghost row has been discarded, causing the crash reported in KUDU-2233. This is a small subset of a broader set of problems that happen when compactions are performed under such snapshots, the most obvious one being that updates are discarded silently. This state is hard to reach as it requires a restart of a tablet server with a considerable amount of data (i.e. not empty) with a single wal segment (previous segments having been gc'd in the previous incarnation), with a single op in it. In particular this single op needs to be a NO_OP or a CHANGE_CONFIG op, as these are the only kinds of operations whose timestamps aren't used to update the clock. In order to get to this and similar states in fuzz-itest this patch adds three new types of operations: - Roll log - Rolls the log if possible. - GC log - Garbage collects log segments, if possible. - Run election - Runs an (emulated) leader election increasing the term and adding a NO_OP to the log. These operations are presently not part of the operations considered when generating random test sequences as those might trigger the bug. This patch adds a narrow, exact repro of the issue that is described in KUDU-2233 in the form of an additional test that is presently disabled. The repro is narrow because it attempts to mimic the crash from the original ticket exactly, instead of trying to catch the broader set of problems. Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c --- M src/kudu/consensus/log.cc M src/kudu/integration-tests/fuzz-itest.cc 2 files changed, 93 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/8885/6 -- To view, visit http://gerrit.cloudera.org:8080/8885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c Gerrit-Change-Number: 8885 Gerrit-PatchSet: 6 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [tools] make kudu CLI auto-detect the replication scheme
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8915 ) Change subject: [tools] make kudu CLI auto-detect the replication scheme .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 Gerrit-Change-Number: 8915 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 09 Jan 2018 01:38:49 + Gerrit-HasComments: No
[kudu-CR] [tools] make kudu CLI auto-detect the replication scheme
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8915 ) Change subject: [tools] make kudu CLI auto-detect the replication scheme .. [tools] make kudu CLI auto-detect the replication scheme Added functionality to allow for automatic detection of appropriate replica move scenario by the 'kudu tablet replica move' sub-command. This is a follow-up for 9120cdd1d288ef6b3e03e2cd8445436712a7d4a9. Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 Reviewed-on: http://gerrit.cloudera.org:8080/8915 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy --- M src/kudu/consensus/consensus.proto M src/kudu/tools/kudu-admin-test.cc M src/kudu/tools/tool_action_tablet.cc M src/kudu/tserver/tablet_service.cc 4 files changed, 61 insertions(+), 48 deletions(-) Approvals: Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 Gerrit-Change-Number: 8915 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] [tablet] Avoid compactions in the past
Hello Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8965 to look at the new patch set (#3). Change subject: [tablet] Avoid compactions in the past .. [tablet] Avoid compactions in the past A previous patch introduced a reproduction for a bug where, in certain cases, mvcc state would not be advanced at all during tablet bootstrap causing compactions to happen in the past. The fix is to make sure that all consensus rounds advance safe time, even NO_OP and CHANGE_CONFIG rounds. This patch adds a fix, enables the failing itest demonstrating the fix works and adds the new operations to the set of ops considered in the generation of random test cases so that we're more likely to catch bugs like this in the future. Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 --- M src/kudu/integration-tests/fuzz-itest.cc M src/kudu/tablet/tablet_bootstrap.cc 2 files changed, 18 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/8965/3 -- To view, visit http://gerrit.cloudera.org:8080/8965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 Gerrit-Change-Number: 8965 Gerrit-PatchSet: 3 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8901 ) Change subject: [master/tserver] enforce re-replication scheme consistency .. [master/tserver] enforce re-replication scheme consistency Don't register a tablet server with master if they run with different tablet replica replacement schemes. Added an integration test to verify the desired behavior. Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Reviewed-on: http://gerrit.cloudera.org:8080/8901 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy --- M src/kudu/consensus/CMakeLists.txt A src/kudu/consensus/replica_management.proto M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/tserver/heartbeater.cc 6 files changed, 182 insertions(+), 15 deletions(-) Approvals: Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 6 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8901 ) Change subject: [master/tserver] enforce re-replication scheme consistency .. Patch Set 5: Code-Review+2 I think this is ready to go, so let's commit this for now. If there are still concerns later about the method of feature detection we can revisit the implementation. -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Tue, 09 Jan 2018 01:35:15 + Gerrit-HasComments: No
[kudu-CR] Add some DVLOG statements to help in debugging compaction issues
Hello Mike Percy, Kudu Jenkins, Andrew Wong, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8886 to look at the new patch set (#5). Change subject: Add some DVLOG statements to help in debugging compaction issues .. Add some DVLOG statements to help in debugging compaction issues This adds some DVLOG statemenst that were crucial in debugging KUDU-2233, particularly aroung merge compactions, clock advancement and log gc. Change-Id: I958045a75feceee19c09961de106bbacd343c739 --- M src/kudu/tablet/compaction.cc M src/kudu/tablet/mvcc.cc M src/kudu/tablet/tablet_replica.cc 3 files changed, 21 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/86/8886/5 -- To view, visit http://gerrit.cloudera.org:8080/8886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I958045a75feceee19c09961de106bbacd343c739 Gerrit-Change-Number: 8886 Gerrit-PatchSet: 5 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [mvcc] Fix watermark advancement in the absence of committed transactions
Hello Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8964 to look at the new patch set (#2). Change subject: [mvcc] Fix watermark advancement in the absence of committed transactions .. [mvcc] Fix watermark advancement in the absence of committed transactions MvccSnapshot's 'none_committed_at_or_after_' (ncaoa) watermark indicates the point after which no transactions are committed. This is used to cull redo deltas that cannot contain any committed transactions under a snapshot, thus reducing work when applying deltas. By definition, this watermark is never supposed to be lower than the 'all_committed_before_' watermark. If this invariant is broken delta selection might be wrong by skipping whole delta files. Normally the ncaoa watermark is advanced when transactions are marked as committed, in which case this is done correctly. However there is problem when the 'all_committed_before_' watermark is advanced and there are no committed or in flight transactions. In this case the ncaoa watermark might be left behind, manifesting as incorrect delta application/skipping or even crashes. This patch includes a fix and a very directed test that makes sure it is correct. A follow up patch includes a new test in fuzz-itest that would trigger a crash without the fix. Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db --- M src/kudu/tablet/mvcc-test.cc M src/kudu/tablet/mvcc.cc M src/kudu/tablet/mvcc.h 3 files changed, 50 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/8964/2 -- To view, visit http://gerrit.cloudera.org:8080/8964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db Gerrit-Change-Number: 8964 Gerrit-PatchSet: 2 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [mvcc] Fix watermark advancement in the absence of committed transactions
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8964 ) Change subject: [mvcc] Fix watermark advancement in the absence of committed transactions .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/8964/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8964/1//COMMIT_MSG@9 PS1, Line 9: ncaof > this abbreviation doesn't seem to match the first letters of those words :) Done http://gerrit.cloudera.org:8080/#/c/8964/1/src/kudu/tablet/mvcc-test.cc File src/kudu/tablet/mvcc-test.cc: http://gerrit.cloudera.org:8080/#/c/8964/1/src/kudu/tablet/mvcc-test.cc@745 PS1, Line 745: // In this case we'll advance safe/clean time but not the 'none_committed_at_or_after_', : // meaning this watermark remains lower than safe/clean time. : / > i'm confused by this comment for this test... are you describing the test's right, my bad. fixed -- To view, visit http://gerrit.cloudera.org:8080/8964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db Gerrit-Change-Number: 8964 Gerrit-PatchSet: 1 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 09 Jan 2018 01:32:49 + Gerrit-HasComments: Yes
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8901 ) Change subject: [master/tserver] enforce re-replication scheme consistency .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto File src/kudu/master/master.proto: http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto@294 PS4, Line 294: optional consensus.ReplicaManagementInfoPB replica_management_info = 7; > After some consideration I decided to go with this approach. The reasoning Alexey and I just spent some time offline investigating the possibility of using feature flags. The biggest reason why feature flags alone don't appear to be sufficient for this use case is that we want to implement different behavior not only based on capabilities but also based on whether some feature is enabled from a configuration perspective. Since we have to implement some application logic on a per-message basis to achieve that, using a feature flag doesn't seem to be sufficient in this case. -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Tue, 09 Jan 2018 01:28:13 + Gerrit-HasComments: Yes
[kudu-CR] Add some DVLOG statements to help in debugging compaction issues
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8886 ) Change subject: Add some DVLOG statements to help in debugging compaction issues .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/8886/4/src/kudu/tablet/compaction.cc File src/kudu/tablet/compaction.cc: http://gerrit.cloudera.org:8080/#/c/8886/4/src/kudu/tablet/compaction.cc@507 PS4, Line 507: 3 > why's this one 3 instead of 4 like the rest? Done -- To view, visit http://gerrit.cloudera.org:8080/8886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I958045a75feceee19c09961de106bbacd343c739 Gerrit-Change-Number: 8886 Gerrit-PatchSet: 4 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 09 Jan 2018 01:27:17 + Gerrit-HasComments: Yes
[kudu-CR] Add some DVLOG statements to help in debugging compaction issues
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8886 ) Change subject: Add some DVLOG statements to help in debugging compaction issues .. Patch Set 4: I've had to rewrite a version of these in some form between 3 and 5 times, so I just figured I should make them permanent. These are DVLOG and a high level at that, so they're precisely to be used in unite test scenarios. -- To view, visit http://gerrit.cloudera.org:8080/8886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I958045a75feceee19c09961de106bbacd343c739 Gerrit-Change-Number: 8886 Gerrit-PatchSet: 4 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 09 Jan 2018 01:26:11 + Gerrit-HasComments: No
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8889 ) Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. [consensus] add unsafe gflag to bypass "safe to evict" logic Replica replacement: added an option to attempt Raft configuration update even in absence of healthy majority of tablet replicas. This option is for testing purposes: as we could see during latest load testing, an attempt to change the configuration under such unusual circumstances allowed for exposing some bugs. In run-time, the option is controlled by the newly introduced flag --raft_attempt_to_replace_replica_without_majority. It affects both 3-4-3 and 3-2-3 replica replacement schemes. Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Reviewed-on: http://gerrit.cloudera.org:8080/8889 Tested-by: Alexey Serbin Reviewed-by: Mike Percy --- M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/master/catalog_manager.cc 7 files changed, 504 insertions(+), 373 deletions(-) Approvals: Alexey Serbin: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 6 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Mike Percy
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8889 ) Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 09 Jan 2018 01:21:43 + Gerrit-HasComments: No
[kudu-CR] KUDU-1097: turn on 3-4-3 replica management scheme
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8862 ) Change subject: KUDU-1097: turn on 3-4-3 replica management scheme .. KUDU-1097: turn on 3-4-3 replica management scheme This patch enables 3-4-3 replica management scheme for tablets: now the 3-4-3 scheme is used by default. To switch back to 3-2-3 scheme, set --raft_prepare_replacement_before_eviction=false for both masters and tablet servers. Change-Id: I781d02fe455140ed3a4693438eece74f07700946 Reviewed-on: http://gerrit.cloudera.org:8080/8862 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy --- M src/kudu/consensus/raft_consensus.cc 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I781d02fe455140ed3a4693438eece74f07700946 Gerrit-Change-Number: 8862 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] [docs] Added steps to update HMS after migrating to multiple Kudu masters
Hello Thomas Tauber-Marshall, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8948 to look at the new patch set (#8). Change subject: [docs] Added steps to update HMS after migrating to multiple Kudu masters .. [docs] Added steps to update HMS after migrating to multiple Kudu masters Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db --- M docs/administration.adoc M docs/security.adoc 2 files changed, 48 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/8948/8 -- To view, visit http://gerrit.cloudera.org:8080/8948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db Gerrit-Change-Number: 8948 Gerrit-PatchSet: 8 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[kudu-CR] KUDU-1097: turn on 3-4-3 replica management scheme
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8862 ) Change subject: KUDU-1097: turn on 3-4-3 replica management scheme .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I781d02fe455140ed3a4693438eece74f07700946 Gerrit-Change-Number: 8862 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Tue, 09 Jan 2018 01:12:36 + Gerrit-HasComments: No
[kudu-CR] [docs] Added steps to update HMS after migrating to multiple Kudu masters
Hello Thomas Tauber-Marshall, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8948 to look at the new patch set (#7). Change subject: [docs] Added steps to update HMS after migrating to multiple Kudu masters .. [docs] Added steps to update HMS after migrating to multiple Kudu masters Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db --- M docs/administration.adoc 1 file changed, 35 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/8948/7 -- To view, visit http://gerrit.cloudera.org:8080/8948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db Gerrit-Change-Number: 8948 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[kudu-CR](branch-1.5.x) Add 'kudu fs list' tool
Hello Kudu Jenkins, Adar Dembo, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/8970 to review the following change. Change subject: Add 'kudu fs list' tool .. Add 'kudu fs list' tool This tool aims to replace exploratory usages of 'kudu fs dump' and 'kudu local_replica dump' with an improved, unified tool. 'kudu fs list' is more flexible, easier to use, and can show more information. Output is formatted using the DataTable abstraction, which gives it good default pretty-printing, with options to output in CSV and JSON for scripts. Results can easily be filtered to a specific table, tablet, column, rowset, or block using flags. The tool can output many different fields: table, table-id, tablet-id, partition, rowset-id, block-id, block-kind, column, column-id, cfile-data-type, cfile-encoding, cfile-compression, cfile-num-values, cfile-size cfile-incompatible-features, cfile-compatible-features, cfile-min-key, cfile-max-key, and cfile-delta-stats. More fields should be straightforward to add. The tool transparently joins information from tablet superblocks with CFile footers, only materializing the metadata necessary to satisfy the requested fields and filters. Examples: To get our bearings, let's look at what tablets are stored on a local tablet server: ```bash $ kudu fs list --fs-wal-dir=/data/kudu/tserver \ --columns="table, table-id, tablet-id, partition" table | table-id |tablet-id |partition ---+--+--+- loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 2a631714f2d243ff92bf525630baa1ec | HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 36827286a00049bc8b242243c6728157 | HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 3880b30ccebd4ede867febd9c7d5580f | HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 39436e9e17d84884b1cb689e88b8415f | HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 44252efb9aaa4c2c963cf6dd5e875c04 | HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 57c92ed8391b4d2bbfdeb339f9fb59fd | HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 68a64aba5917499ebb7773f16bcd6f6d | HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 6b5f0729a9bf454791239f77b0912f4e | HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 8a2d120bd6984144ae963bfe8435206e | HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 8b3ba4f415f945849a6a690a142cf1e4 | HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 9656be3aa07248a69e3ad6edaa0048cb | HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 9e8e444d079842a9b4a83ee9f8bed633 | HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | a794a8e5d3f24e70a96b0beb5a355823 | HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | bfb8f24b91cd4ecf924aacbb37125041 | HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED foo | e184a99893b44b17a7b2131123c6de0e | c3ce418c72ab4fea8548387f236dd1fa | loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | e00a284081ca468a994a3609a511e886 | HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | efa22fc899a44bb2a16f620464a15c60 | HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED ``` The 'foo' table looks interesting; let's drill down into its tablet, and see what rowsets and blocks it has, and some of their associated metadata: ```bash $ kudu fs list --fs-wal-dir=/data/kudu/tserver \ --
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/8962 )
Change subject: consensus: Reduce copy/paste for observer callbacks
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8962/1/src/kudu/consensus/consensus_queue.cc
File src/kudu/consensus/consensus_queue.cc:
http://gerrit.cloudera.org:8080/#/c/8962/1/src/kudu/consensus/consensus_queue.cc@1323
PS1, Line 1323: void
PeerMessageQueue::NotifyObserversTask(std::function
func) {
> warning: the parameter 'func' is copied for each invocation but only used a
seems accurate
--
To view, visit http://gerrit.cloudera.org:8080/8962
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437
Gerrit-Change-Number: 8962
Gerrit-PatchSet: 1
Gerrit-Owner: Mike Percy
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 09 Jan 2018 00:10:21 +
Gerrit-HasComments: Yes
[kudu-CR] [mvcc] Fix watermark advancement in the absence of committed transactions
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8964 ) Change subject: [mvcc] Fix watermark advancement in the absence of committed transactions .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/8964/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8964/1//COMMIT_MSG@9 PS1, Line 9: ncaof this abbreviation doesn't seem to match the first letters of those words :) http://gerrit.cloudera.org:8080/#/c/8964/1/src/kudu/tablet/mvcc-test.cc File src/kudu/tablet/mvcc-test.cc: http://gerrit.cloudera.org:8080/#/c/8964/1/src/kudu/tablet/mvcc-test.cc@745 PS1, Line 745: // In this case we'll advance safe/clean time but not the 'none_committed_at_or_after_', : // meaning this watermark remains lower than safe/clean time. : / i'm confused by this comment for this test... are you describing the test's status when it was failing? -- To view, visit http://gerrit.cloudera.org:8080/8964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db Gerrit-Change-Number: 8964 Gerrit-PatchSet: 1 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 09 Jan 2018 00:03:10 + Gerrit-HasComments: Yes
[kudu-CR] Add some DVLOG statements to help in debugging compaction issues
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8886 ) Change subject: Add some DVLOG statements to help in debugging compaction issues .. Patch Set 4: (1 comment) seems fine, but do you think these are useful enough to commit or were they one-offs for debugging this issue? it seems like enabling them outside of a unit test scenario would probably be so noisy as to be not useful http://gerrit.cloudera.org:8080/#/c/8886/4/src/kudu/tablet/compaction.cc File src/kudu/tablet/compaction.cc: http://gerrit.cloudera.org:8080/#/c/8886/4/src/kudu/tablet/compaction.cc@507 PS4, Line 507: 3 why's this one 3 instead of 4 like the rest? -- To view, visit http://gerrit.cloudera.org:8080/8886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I958045a75feceee19c09961de106bbacd343c739 Gerrit-Change-Number: 8886 Gerrit-PatchSet: 4 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Jan 2018 23:59:20 + Gerrit-HasComments: Yes
[kudu-CR] [docs] Added a section for Kudu clients connecting to secure clusters
Hello Alexey Serbin, Dan Burkert, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8953 to look at the new patch set (#8). Change subject: [docs] Added a section for Kudu clients connecting to secure clusters .. [docs] Added a section for Kudu clients connecting to secure clusters Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 --- M docs/security.adoc 1 file changed, 13 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/8953/8 -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 8 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/8951 ) Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo .. Patch Set 6: Without profiling it I don't have a great explanation for why TSAN is particularly slow on this workload. It's definitely slower on many workloads though, so increasing timeout in TSAN is probably reasonable if you don't want to take the time to profile it. -- To view, visit http://gerrit.cloudera.org:8080/8951 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb Gerrit-Change-Number: 8951 Gerrit-PatchSet: 6 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Jan 2018 23:56:54 + Gerrit-HasComments: No
[kudu-CR] [docs] Added additional steps for updating HMS when migrating to multiple Kudu masters
Hello Alexey Serbin, Dan Burkert, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8953 to look at the new patch set (#7). Change subject: [docs] Added additional steps for updating HMS when migrating to multiple Kudu masters .. [docs] Added additional steps for updating HMS when migrating to multiple Kudu masters Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 --- M docs/security.adoc 1 file changed, 13 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/8953/7 -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [fuzz-itest] Add the ability to roll/gc logs and to run elections
David Ribeiro Alves has abandoned this change. ( http://gerrit.cloudera.org:8080/8884 ) Change subject: [fuzz-itest] Add the ability to roll/gc logs and to run elections .. Abandoned merged this with another patch -- To view, visit http://gerrit.cloudera.org:8080/8884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I3708366fcd941098e784d911676a4deb32c67c50 Gerrit-Change-Number: 8884 Gerrit-PatchSet: 7 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] [tablet] Avoid compactions in the past
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8965 to look at the new patch set (#2). Change subject: [tablet] Avoid compactions in the past .. [tablet] Avoid compactions in the past A previous patch introduced a reproduction for a bug where, in certain cases, mvcc state would not be advanced at all during tablet bootstrap causing compactions to happen in the past. The fix is to make sure that all consensus rounds advance safe time, even NO_OP and CHANGE_CONFIG rounds. This patch adds a fix, enables the failing itest demonstrating the fix works and adds the new operations to the set of ops considered in the generation of random test cases so that we're more likely to catch bugs like this in the future. Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 --- M src/kudu/integration-tests/fuzz-itest.cc M src/kudu/tablet/tablet_bootstrap.cc 2 files changed, 12 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/8965/2 -- To view, visit http://gerrit.cloudera.org:8080/8965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 Gerrit-Change-Number: 8965 Gerrit-PatchSet: 2 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2233 Add a test case for compactions in the past
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8885 to look at the new patch set (#4). Change subject: KUDU-2233 Add a test case for compactions in the past .. KUDU-2233 Add a test case for compactions in the past In some cases we're performing compactions before the clock has been updated at all, making the compaction snapshot an empty one. This makes it so the compaction runs on a snapshot that discards all redo deltas and applies all undos. When a compaction with such a snapshot finds two duplicate rows (a ghost and live row) it can't distinguish the two because the redo DELETE of the ghost row has been discarded, causing the crash reported in KUDU-2233. This is a small subset of a broader set of problems that happen when compactions are performed under such snapshots, the most obvious one being that updates are discarded silently. This state is hard to reach as it requires a restart of a tablet server with a considerable amount of data (i.e. not empty) with a single wal segment (previous segments having been gc'd in the previous incarnation), with a single op in it. In particular this single op needs to be a NO_OP or a CHANGE_CONFIG op, as these are the only kinds of operations whose timestamps aren't used to update the clock. In order to get to this and similar states in fuzz-itest this patch adds three new types of operations: - Roll log - Rolls the log if possible. - GC log - Garbage collects log segments, if possible. - Run election - Runs an (emulated) leader election increasing the term and adding a NO_OP to the log. These operations are presently not part of the operations considered when generating random test sequences as those might trigger the bug. This patch adds a narrow, exact repro of the issue that is described in KUDU-2233 in the form of an additional test that is presently disabled. The repro is narrow because it attempts to mimic the crash from the original ticket exactly, instead of trying to catch the broader set of problems. Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c --- M src/kudu/consensus/log.cc M src/kudu/integration-tests/fuzz-itest.cc 2 files changed, 89 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/8885/4 -- To view, visit http://gerrit.cloudera.org:8080/8885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c Gerrit-Change-Number: 8885 Gerrit-PatchSet: 4 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8901 ) Change subject: [master/tserver] enforce re-replication scheme consistency .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto File src/kudu/master/master.proto: http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto@294 PS4, Line 294: optional consensus.ReplicaManagementInfoPB replica_management_info = 7; > I thought about the feature flag, but I decided to go with a field. After some consideration I decided to go with this approach. The reasoning: 1. The feature flags, introduced at this point, are not going to help with this particular feature since the older version don't know about would-be-introduced feature flags yet. 2. We have to propagate the same information to the Kudu CLI tool to allow automatic detection of the replica management scheme (see https://gerrit.cloudera.org/#/c/8915/), so there we re-use the consensus.ReplicaManagementInfoPB structure. If we want to have a feature flags here anyway, we can add those in a separate commit, after some quick design review. -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 08 Jan 2018 23:50:40 + Gerrit-HasComments: Yes
[kudu-CR](branch-1.6.x) Add 'kudu fs list' tool
Dan Burkert has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8967 Change subject: Add 'kudu fs list' tool .. Add 'kudu fs list' tool This tool aims to replace exploratory usages of 'kudu fs dump' and 'kudu local_replica dump' with an improved, unified tool. 'kudu fs list' is more flexible, easier to use, and can show more information. Output is formatted using the DataTable abstraction, which gives it good default pretty-printing, with options to output in CSV and JSON for scripts. Results can easily be filtered to a specific table, tablet, column, rowset, or block using flags. The tool can output many different fields: table, table-id, tablet-id, partition, rowset-id, block-id, block-kind, column, column-id, cfile-data-type, cfile-encoding, cfile-compression, cfile-num-values, cfile-size cfile-incompatible-features, cfile-compatible-features, cfile-min-key, cfile-max-key, and cfile-delta-stats. More fields should be straightforward to add. The tool transparently joins information from tablet superblocks with CFile footers, only materializing the metadata necessary to satisfy the requested fields and filters. Examples: To get our bearings, let's look at what tablets are stored on a local tablet server: ```bash $ kudu fs list --fs-wal-dir=/data/kudu/tserver \ --columns="table, table-id, tablet-id, partition" table | table-id |tablet-id |partition ---+--+--+- loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 2a631714f2d243ff92bf525630baa1ec | HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 36827286a00049bc8b242243c6728157 | HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 3880b30ccebd4ede867febd9c7d5580f | HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 39436e9e17d84884b1cb689e88b8415f | HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 44252efb9aaa4c2c963cf6dd5e875c04 | HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 57c92ed8391b4d2bbfdeb339f9fb59fd | HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 68a64aba5917499ebb7773f16bcd6f6d | HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 6b5f0729a9bf454791239f77b0912f4e | HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | 8a2d120bd6984144ae963bfe8435206e | HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 8b3ba4f415f945849a6a690a142cf1e4 | HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 9656be3aa07248a69e3ad6edaa0048cb | HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | 9e8e444d079842a9b4a83ee9f8bed633 | HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | a794a8e5d3f24e70a96b0beb5a355823 | HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | bfb8f24b91cd4ecf924aacbb37125041 | HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED foo | e184a99893b44b17a7b2131123c6de0e | c3ce418c72ab4fea8548387f236dd1fa | loadgen_auto_fa03aaf7bdf54bb4896c534f38d177a1 | 800af247c1424ecd8e96a37b5ee4d311 | e00a284081ca468a994a3609a511e886 | HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED loadgen_auto_06c8038c02da40048397e4f6ad1662c3 | 84ff589b979e4f90aa630e7179fcb644 | efa22fc899a44bb2a16f620464a15c60 | HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED ``` The 'foo' table looks interesting; let's drill down into its tablet, and see what rowsets and blocks it has, and some of their associated metadata: ```bash $ kudu fs list --fs-wal-dir=/data/kudu/tserver \ --columns="rowset-id, column, column-id, block-kind, block-id" \
[kudu-CR] [tools] make kudu CLI auto-detect the replication scheme
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8915 ) Change subject: [tools] make kudu CLI auto-detect the replication scheme .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/8915/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8915/3//COMMIT_MSG@7 PS3, Line 7: [tools] kudu CLI auto detects replication scheme > nit: how about: That seems a bit long. I resorted to: [tools] make kudu CLI auto-detect the replication scheme http://gerrit.cloudera.org:8080/#/c/8915/3/src/kudu/consensus/consensus.proto File src/kudu/consensus/consensus.proto: http://gerrit.cloudera.org:8080/#/c/8915/3/src/kudu/consensus/consensus.proto@496 PS3, Line 496: optional ReplicaManagementInfoPB replica_management_info = 3; > nit: we usually leave the error as the last entry in the message, even if t Done -- To view, visit http://gerrit.cloudera.org:8080/8915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 Gerrit-Change-Number: 8915 Gerrit-PatchSet: 3 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Mon, 08 Jan 2018 23:42:16 + Gerrit-HasComments: Yes
[kudu-CR] [tools] make kudu CLI auto-detect the replication scheme
Hello Mike Percy, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8915 to look at the new patch set (#4). Change subject: [tools] make kudu CLI auto-detect the replication scheme .. [tools] make kudu CLI auto-detect the replication scheme Added functionality to allow for automatic detection of appropriate replica move scenario by the 'kudu tablet replica move' sub-command. This is a follow-up for 9120cdd1d288ef6b3e03e2cd8445436712a7d4a9. Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 --- M src/kudu/consensus/consensus.proto M src/kudu/tools/kudu-admin-test.cc M src/kudu/tools/tool_action_tablet.cc M src/kudu/tserver/tablet_service.cc 4 files changed, 61 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/8915/4 -- To view, visit http://gerrit.cloudera.org:8080/8915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia536c1f64ef8173aa4385db8ba01581bc3528154 Gerrit-Change-Number: 8915 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] [docs] Added steps to update HMS after migrating to multiple Kudu masters
Hello Thomas Tauber-Marshall, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8948 to look at the new patch set (#5). Change subject: [docs] Added steps to update HMS after migrating to multiple Kudu masters .. [docs] Added steps to update HMS after migrating to multiple Kudu masters Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db --- M docs/administration.adoc 1 file changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/8948/5 -- To view, visit http://gerrit.cloudera.org:8080/8948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db Gerrit-Change-Number: 8948 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[kudu-CR] [docs] Added additional steps for migrating to multiple Kudu masters
Hello Alexey Serbin, Dan Burkert, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8953 to look at the new patch set (#6). Change subject: [docs] Added additional steps for migrating to multiple Kudu masters .. [docs] Added additional steps for migrating to multiple Kudu masters Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 --- M docs/administration.adoc M docs/security.adoc 2 files changed, 33 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/8953/6 -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8901 ) Change subject: [master/tserver] enforce re-replication scheme consistency .. Patch Set 4: (9 comments) http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto File src/kudu/master/master.proto: http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto@338 PS4, Line 338: fields > field Done http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master.proto@340 PS4, Line 340: optional consensus.ReplicaManagementInfoPB replica_management_info = 10; > is this ever used? if not, let's drop it Good catch -- it seems this is just a left-over. http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc File src/kudu/master/master_service.cc: http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@171 PS4, Line 171: $0 > ($0) Done http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@172 PS4, Line 172: $3 > ($3) Done http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@186 PS4, Line 186: return rpc->RespondSuccess(); > nit: how about It's two times more lines :) http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@194 PS4, Line 194: unknown > It's better to ignore tidy bot's warning than to add "unknown", IMHO. Done http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@195 PS4, Line 195: return rpc->RespondFailure(s); > nit: hmm, maybe this is personal preference but return x seems confusing in In C++ that's legal: it returns void from a function which has 'void' as a return type. All right -- I'll return back the two-line version. http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/master/master_service.cc@258 PS4, Line 258: return rpc->RespondSuccess(); > nit: "return" at the end of a void function seems confusing and there is re This was an unification effort. All right, I'll remove this. http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/tserver/heartbeater.cc File src/kudu/tserver/heartbeater.cc: http://gerrit.cloudera.org:8080/#/c/8901/4/src/kudu/tserver/heartbeater.cc@460 PS4, Line 460: return StatusFromPB(error->status()); > nit: this repeats the line below. maybe just don't swap above, but copy? I resorted to return StatusFromPB(error ? error->status() : resp.error().status()); -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 08 Jan 2018 23:35:15 + Gerrit-HasComments: Yes
[kudu-CR] [master/tserver] enforce re-replication scheme consistency
Hello Will Berkeley, Tidy Bot, Mike Percy, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8901 to look at the new patch set (#5). Change subject: [master/tserver] enforce re-replication scheme consistency .. [master/tserver] enforce re-replication scheme consistency Don't register a tablet server with master if they run with different tablet replica replacement schemes. Added an integration test to verify the desired behavior. Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 --- M src/kudu/consensus/CMakeLists.txt A src/kudu/consensus/replica_management.proto M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/tserver/heartbeater.cc 6 files changed, 182 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/8901/5 -- To view, visit http://gerrit.cloudera.org:8080/8901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I71c4c2e72bb2d62cec6de0f6d00b418377e8ae85 Gerrit-Change-Number: 8901 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley
[kudu-CR] [doc] Added a section for Kudu clients connecting to secure clusters.
Jean-Daniel Cryans has posted comments on this change. ( http://gerrit.cloudera.org:8080/8953 ) Change subject: [doc] Added a section for Kudu clients connecting to secure clusters. .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/8953/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8953/5//COMMIT_MSG@7 PS5, Line 7: s. nit: don't end the first line in a commit message with a period. -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 22:39:22 + Gerrit-HasComments: Yes
[kudu-CR] [docs] Added steps to update HMS after migrating to multiple Kudu masters.
Jean-Daniel Cryans has posted comments on this change. ( http://gerrit.cloudera.org:8080/8948 ) Change subject: [docs] Added steps to update HMS after migrating to multiple Kudu masters. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/8948/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8948/4//COMMIT_MSG@7 PS4, Line 7: . nit: don't end the first line in a commit message with a period. -- To view, visit http://gerrit.cloudera.org:8080/8948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db Gerrit-Change-Number: 8948 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 08 Jan 2018 22:38:36 + Gerrit-HasComments: Yes
[kudu-CR] [docs] Added steps to update HMS after migrating to multiple Kudu masters.
Jean-Daniel Cryans has posted comments on this change. ( http://gerrit.cloudera.org:8080/8948 ) Change subject: [docs] Added steps to update HMS after migrating to multiple Kudu masters. .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/8948/4/docs/administration.adoc File docs/administration.adoc: http://gerrit.cloudera.org:8080/#/c/8948/4/docs/administration.adoc@261 PS4, Line 261: HMS Given that this is the first time we talk about the HMS, maybe give the full name first? http://gerrit.cloudera.org:8080/#/c/8948/4/docs/administration.adoc@262 PS4, Line 262: * If you set up the DNS aliases, use the following statement, replacing `master-1`, `master-2`, and `master-3` with your actual aliases. Even in docs we try to no go beyond 100 characters per line. Mind fixing this here and below? -- To view, visit http://gerrit.cloudera.org:8080/8948 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab3999c9e581ed3591b220c08491cdae867c91db Gerrit-Change-Number: 8948 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 08 Jan 2018 22:37:51 + Gerrit-HasComments: Yes
[kudu-CR] [doc] Added a section for Kudu clients connecting to secure clusters.
Hello Alexey Serbin, Dan Burkert, Jean-Daniel Cryans, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8953 to look at the new patch set (#5). Change subject: [doc] Added a section for Kudu clients connecting to secure clusters. .. [doc] Added a section for Kudu clients connecting to secure clusters. Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 --- M docs/security.adoc 1 file changed, 15 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/53/8953/5 -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [doc] Added a section for Kudu clients connecting to secure clusters.
arod...@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/8953 ) Change subject: [doc] Added a section for Kudu clients connecting to secure clusters. .. Patch Set 4: > (1 comment) Updated. -- To view, visit http://gerrit.cloudera.org:8080/8953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabee52216b4db0e18e5f693375f0a3e0f0a8a164 Gerrit-Change-Number: 8953 Gerrit-PatchSet: 4 Gerrit-Owner: arod...@cloudera.com Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: arod...@cloudera.com Gerrit-Comment-Date: Mon, 08 Jan 2018 21:59:14 + Gerrit-HasComments: No
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8962 ) Change subject: consensus: Reduce copy/paste for observer callbacks .. Patch Set 1: Code-Review+1 LGTM, maybe just fix what IWYU reports. -- To view, visit http://gerrit.cloudera.org:8080/8962 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Gerrit-Change-Number: 8962 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Mon, 08 Jan 2018 21:39:31 + Gerrit-HasComments: No
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8951 ) Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8951 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb Gerrit-Change-Number: 8951 Gerrit-PatchSet: 6 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Jan 2018 21:34:30 + Gerrit-HasComments: No
[kudu-CR] [raft] Add an EmulateElection variant that waits for one more commit
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8883 ) Change subject: [raft] Add an EmulateElection variant that waits for one more commit .. Patch Set 2: fair point, abandoning this one -- To view, visit http://gerrit.cloudera.org:8080/8883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4815abf09d298f227ffe57b68d5b429c43c1d69b Gerrit-Change-Number: 8883 Gerrit-PatchSet: 2 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Mon, 08 Jan 2018 20:55:54 + Gerrit-HasComments: No
[kudu-CR] [raft] Add an EmulateElection variant that waits for one more commit
David Ribeiro Alves has abandoned this change. ( http://gerrit.cloudera.org:8080/8883 ) Change subject: [raft] Add an EmulateElection variant that waits for one more commit .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/8883 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I4815abf09d298f227ffe57b68d5b429c43c1d69b Gerrit-Change-Number: 8883 Gerrit-PatchSet: 2 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] [mvcc] Fix watermark advancement in the absence of committed transactions
David Ribeiro Alves has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8964 Change subject: [mvcc] Fix watermark advancement in the absence of committed transactions .. [mvcc] Fix watermark advancement in the absence of committed transactions MvccSnapshot's 'none_committed_at_or_after_' (ncaof) watermark indicates the point after which no transactions are committed. This is used to cull redo deltas that cannot contain any committed transactions under a snapshot, thus reducing work when applying deltas. By definition, this watermark is never supposed to be lower than the 'all_committed_before_' watermark. If this invariant is broken delta selection might be wrong by skipping whole delta files. Normally the ncaof watermark is advanced when transactions are marked as committed, in which case this is done correctly. However there is problem when the 'all_committed_before_' watermark is advanced and there are no committed or in flight transactions. In this case the ncaof watermark might be left behind, manifesting as incorrect delta application/skipping or even crashes. This patch includes a fix and a very directed test that makes sure it is correct. A follow up patch includes a new test in fuzz-itest that would trigger a crash without the fix. Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db --- M src/kudu/tablet/mvcc-test.cc M src/kudu/tablet/mvcc.cc M src/kudu/tablet/mvcc.h 3 files changed, 48 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/8964/1 -- To view, visit http://gerrit.cloudera.org:8080/8964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icf94ea289204adf2eed797c3623cdea1b11b09db Gerrit-Change-Number: 8964 Gerrit-PatchSet: 1 Gerrit-Owner: David Ribeiro Alves
[kudu-CR] KUDU-2233 Add a test case for compactions in the past
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8885 to look at the new patch set (#3). Change subject: KUDU-2233 Add a test case for compactions in the past .. KUDU-2233 Add a test case for compactions in the past In some cases we're performing compactions before the clock has been updated at all, making the compaction snapshot an empty one. This makes it so the compaction runs on a snapshot that discards all redo deltas and applies all undos. When a compaction with such a snapshot finds two duplicate rows (a ghost and live row) it can't distinguish the two because the redo DELETE of the ghost row has been discarded, causing the crash reported in KUDU-2233. This is a small subset of a broader set of problems that happen when compactions are performed under such snapshots, the most obvious one being that updates are discarded silently. This state is hard to reach as it requires a restart of a tablet server with a considerable amount of data (i.e. not empty) with a single wal segment (previous segments having been gc'd in the previous incarnation), with a single op in it. In particular this single op needs to be a NO_OP or a CHANGE_CONFIG op, as these are the only kinds of operations whose timestamps aren't used to update the clock. This patch adds a narrow, exact repro of the issue that is described in KUDU-2233. The repro is narrow because it attempts to mimic the crash from the original ticket exactly, instead of trying to catch the broader set of problems. Follow up patches will add broader checks and a fix. Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c --- M src/kudu/integration-tests/fuzz-itest.cc 1 file changed, 19 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/8885/3 -- To view, visit http://gerrit.cloudera.org:8080/8885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibf5685897ef7580dd743faac4d68690a25663e4c Gerrit-Change-Number: 8885 Gerrit-PatchSet: 3 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [fuzz-itest] Add the ability to roll/gc logs and to run elections
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8884 to look at the new patch set (#7). Change subject: [fuzz-itest] Add the ability to roll/gc logs and to run elections .. [fuzz-itest] Add the ability to roll/gc logs and to run elections There are certain bugs that depend on what is present in the WAL at boostrap time. Just recently, we found KUDU-2233, a bug that only manifested when the tablet server booted to a WAL that was devoid of entries other than NO_OPs of CONFIG_CHANGEs. To make sure that these cases also have a chance to occur in fuzz-itest this patch add three new types of operations: - Roll log - Rolls the log if possible. - GC log - Garbage collects log segments, if possible. - Run election - Runs an (emulated) leader election increasing the term and adding a NO_OP to the log. Change-Id: I3708366fcd941098e784d911676a4deb32c67c50 --- M src/kudu/consensus/log.cc M src/kudu/integration-tests/fuzz-itest.cc 2 files changed, 69 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/84/8884/7 -- To view, visit http://gerrit.cloudera.org:8080/8884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3708366fcd941098e784d911676a4deb32c67c50 Gerrit-Change-Number: 8884 Gerrit-PatchSet: 7 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] [tablet] Add a pre-compaction check to avoid compactions in the past
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8887 to look at the new patch set (#3). Change subject: [tablet] Add a pre-compaction check to avoid compactions in the past .. [tablet] Add a pre-compaction check to avoid compactions in the past This adds a pre-compaction check that gets the max timestamp of all the rows that will be considered for compaction and makes sure that timestamp is considered committed under the compaction snapshot. This is generalization of the repro introduced in previous patches for KUDU-2233 but that now applies to all compactions in the past and not only those which have duplicated rows in the inputs. This is not a fix yet, that's left for follow up patches. Change-Id: I80edc9dc8d5eec475998a9c721b9f591eb6180dc --- M src/kudu/tablet/compaction.cc M src/kudu/tablet/compaction.h M src/kudu/tablet/delta_tracker.cc M src/kudu/tablet/delta_tracker.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset.cc M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/rowset.cc M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.cc 12 files changed, 153 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/87/8887/3 -- To view, visit http://gerrit.cloudera.org:8080/8887 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I80edc9dc8d5eec475998a9c721b9f591eb6180dc Gerrit-Change-Number: 8887 Gerrit-PatchSet: 3 Gerrit-Owner: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [tablet] Avoid compactions in the past
David Ribeiro Alves has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8965 Change subject: [tablet] Avoid compactions in the past .. [tablet] Avoid compactions in the past A previous patch introduced a reproduction for a bug where, in certain cases, mvcc state would not be advanced at all during tablet bootstrap causing compactions to happen in the past. The fix is to make sure that all consensus rounds advance safe time, even NO_OP and CHANGE_CONFIG rounds. This patch does that and enables the failing itest demonstrating the fix works. Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 --- M src/kudu/integration-tests/fuzz-itest.cc M src/kudu/tablet/tablet_bootstrap.cc 2 files changed, 4 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/8965/1 -- To view, visit http://gerrit.cloudera.org:8080/8965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6db9394d37b4ad1e7b1b1bc6ba0535558c20d9a8 Gerrit-Change-Number: 8965 Gerrit-PatchSet: 1 Gerrit-Owner: David Ribeiro Alves
[kudu-CR] mini cluster: Remove tablet server index by uuid() from interface
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8963 ) Change subject: mini_cluster: Remove tablet_server_index_by_uuid() from interface .. Patch Set 1: Code-Review+1 Looks good to me; but it seems IWYU has some warnings. -- To view, visit http://gerrit.cloudera.org:8080/8963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab Gerrit-Change-Number: 8963 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 20:33:08 + Gerrit-HasComments: No
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8889 ) Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. Patch Set 5: Verified+1 Unrelated flake in MultiThreadedHybridClockTabletTest/3.UpdateNoMergeCompaction -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Mon, 08 Jan 2018 20:26:10 + Gerrit-HasComments: No
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Alexey Serbin has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/8889 ) Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. Removed reviewer Kudu Jenkins with the following votes: * Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Mike Percy
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8951 ) Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo .. Patch Set 6: I changed the asserts and expects to checks, and it didn't have much of an effect. The slowness appears to be specific to TSAN. I'm able to repro the scan timing out (120s+) on a fast machine (va1022), whereas the scans take 2s and 7s with release and debug builds, respectively. Any thoughts on why TSAN may be slow in particular? Next I'm going to verify that the scans are actually completing given a much longer timeout. -- To view, visit http://gerrit.cloudera.org:8080/8951 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb Gerrit-Change-Number: 8951 Gerrit-PatchSet: 6 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Jan 2018 20:07:09 + Gerrit-HasComments: No
[kudu-CR] [java client] update propagated TS for AUTO FLUSH SYNC flush mode
Hao Hao has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8837 ) Change subject: [java client] update propagated TS for AUTO_FLUSH_SYNC flush mode .. [java client] update propagated TS for AUTO_FLUSH_SYNC flush mode Commit be6b81057 updated propagated timestamp for all write modes except AUTO_FLUSH_SYNC for the java client. This patch adds timestamp propagation for this mode and updates TestHybridTime to ensure propagated timestamp is updated for all flush modes. Change-Id: Ibf0ca58b10842cb15ed5db7bcd4694c4d8cc3a89 Reviewed-on: http://gerrit.cloudera.org:8080/8837 Tested-by: Kudu Jenkins Reviewed-by: David Ribeiro Alves --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHybridTime.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScannerMultiTablet.java 5 files changed, 48 insertions(+), 12 deletions(-) Approvals: Kudu Jenkins: Verified David Ribeiro Alves: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibf0ca58b10842cb15ed5db7bcd4694c4d8cc3a89 Gerrit-Change-Number: 8837 Gerrit-PatchSet: 7 Gerrit-Owner: Hao Hao Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] consensus: Reduce copy/paste for observer callbacks
Hello Alexey Serbin, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/8962 to review the following change. Change subject: consensus: Reduce copy/paste for observer callbacks .. consensus: Reduce copy/paste for observer callbacks This is a net-negative line patch that unifies how the PeerMessageQueue notifies PeerMessageQueueObserver instances that are registered with it. The patch removes most of the copy/paste in this part of the code. This patch has no functional changes. Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 --- M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/consensus_queue.h 2 files changed, 33 insertions(+), 84 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/8962/1 -- To view, visit http://gerrit.cloudera.org:8080/8962 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icbc4cb9d7b6e51a9f64d6f08c2f48d89705f5437 Gerrit-Change-Number: 8962 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/8951 )
Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/integration-tests/heavy-update-compaction-itest.cc
File src/kudu/integration-tests/heavy-update-compaction-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/integration-tests/heavy-update-compaction-itest.cc@206
PS6, Line 206: // Scan the updated rows and ensure the final values are
present.
: KuduScanner scanner(table_.get());
: ASSERT_OK(scanner.SetFaultTolerant());
: ASSERT_OK(scanner.AddConjunctPredicate(
: table_->NewComparisonPredicate(
: "key", KuduPredicate::LESS,
KuduValue::FromInt(FLAGS_rows;
:
: // Walking the updates can take a long time.
: scanner.SetTimeoutMillis(120 * 1000);
:
: LOG_TIMING(INFO, "scanning") {
: ASSERT_OK(scanner.Open());
: vector rows;
: size_t final_values_offset = 0;
: while (scanner.HasMoreRows()) {
: ASSERT_OK(scanner.NextBatch(&rows));
: for (const auto & row : rows) {
: for (int idx = 1; idx <= 5; idx++) {
: ASSERT_GT(final_values.size(), final_values_offset);
: Slice actual_val;
: ASSERT_OK(row.GetString(idx, &actual_val));
: EXPECT_EQ(actual_val,
final_values[final_values_offset++]);
: }
: }
: }
: }
> Thanks for doing this. I'm surprised that it might take so long. I guess th
i've also seen ASSERT_OK and EXPECT_* etc be slow in gtest, since it actually
collects all the successes and not just failures. We have ASSERT_OK_FAST which
is a faster version of ASSERT_OK but I don't think there is any such for _EQ.
We could just use CHECK instead though and it might be faster?
--
To view, visit http://gerrit.cloudera.org:8080/8951
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb
Gerrit-Change-Number: 8951
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Burkert
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 08 Jan 2018 19:50:06 +
Gerrit-HasComments: Yes
[kudu-CR] mini cluster: Remove tablet server index by uuid() from interface
Hello Alexey Serbin, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/8963 to review the following change. Change subject: mini_cluster: Remove tablet_server_index_by_uuid() from interface .. mini_cluster: Remove tablet_server_index_by_uuid() from interface The tablet_server_index_by_uuid() method, which common to the InternalMiniCluster and ExternalMiniCluster, isn't really useful from the parent interface (MiniCluster) and isn't compatible with a MixedMiniCluster that I'm working on. Let's just remove it from the pure virtual parent interface. Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab --- M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.h 3 files changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/8963/1 -- To view, visit http://gerrit.cloudera.org:8080/8963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I67d25278ee8c2b9379ed83008f162d8ac36271ab Gerrit-Change-Number: 8963 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8889 ) Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/8889/4/src/kudu/consensus/quorum_util-test.cc File src/kudu/consensus/quorum_util-test.cc: http://gerrit.cloudera.org:8080/#/c/8889/4/src/kudu/consensus/quorum_util-test.cc@a454 PS4, Line 454: : : : : : : : : > Was this test removed? Woops, it's good catch -- I remember removing some duplicate scenario, but apparently it's not a duplicate (the value of the PROMOTE attribute is different from the case above). http://gerrit.cloudera.org:8080/#/c/8889/4/src/kudu/consensus/quorum_util-test.cc@834 PS4, Line 834: : > nit: missing space before colon Done http://gerrit.cloudera.org:8080/#/c/8889/4/src/kudu/consensus/quorum_util.h File src/kudu/consensus/quorum_util.h: http://gerrit.cloudera.org:8080/#/c/8889/4/src/kudu/consensus/quorum_util.h@40 PS4, Line 40: // (this applies to both the current and the result configuration). > I don't think this is true, it should only apply to the resulting configura Yep, it seems that was a brainfart. Sure, that's true only for the resulting config. -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Mon, 08 Jan 2018 19:42:06 + Gerrit-HasComments: Yes
[kudu-CR] [consensus] add unsafe gflag to bypass "safe to evict" logic
Hello Mike Percy, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8889 to look at the new patch set (#5). Change subject: [consensus] add unsafe gflag to bypass "safe to evict" logic .. [consensus] add unsafe gflag to bypass "safe to evict" logic Replica replacement: added an option to attempt Raft configuration update even in absence of healthy majority of tablet replicas. This option is for testing purposes: as we could see during latest load testing, an attempt to change the configuration under such unusual circumstances allowed for exposing some bugs. In run-time, the option is controlled by the newly introduced flag --raft_attempt_to_replace_replica_without_majority. It affects both 3-4-3 and 3-2-3 replica replacement schemes. Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 --- M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/master/catalog_manager.cc 7 files changed, 504 insertions(+), 373 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/8889/5 -- To view, visit http://gerrit.cloudera.org:8080/8889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4f89b6c584296e3da5047475c5c86c4cb1118ad0 Gerrit-Change-Number: 8889 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] consensus: Fix NON VOTER ack-counting bug
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8868 ) Change subject: consensus: Fix NON_VOTER ack-counting bug .. consensus: Fix NON_VOTER ack-counting bug This patch fixes an issue where we were not differentiating between replicating to voters and non-voters. This enables the test written by Alexey and also makes some changes to it. The test fails without this patch and passes with the patch. Tests added: * Added a unit test in consensus_queue-test * Updated and enabled the system test in raft_consensus_nonvoter-itest Change-Id: I13143e9bb4b76af3fd6dada28fcec05b27d24476 Reviewed-on: http://gerrit.cloudera.org:8080/8868 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M src/kudu/consensus/consensus-test-util.h M src/kudu/consensus/consensus_peers-test.cc M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_queue-test.cc M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/consensus_queue.h M src/kudu/consensus/raft_consensus.cc M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc 8 files changed, 233 insertions(+), 88 deletions(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I13143e9bb4b76af3fd6dada28fcec05b27d24476 Gerrit-Change-Number: 8868 Gerrit-PatchSet: 7 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot
[kudu-CR] [consensus] add test for non-voters in consensus queue
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/8867 ) Change subject: [consensus] add test for non-voters in consensus queue .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8867 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16382b75256a32d695d8bc9ce020f40276114511 Gerrit-Change-Number: 8867 Gerrit-PatchSet: 3 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Comment-Date: Mon, 08 Jan 2018 19:37:58 + Gerrit-HasComments: No
[kudu-CR] [consensus] add test for non-voters in consensus queue
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8867 ) Change subject: [consensus] add test for non-voters in consensus queue .. [consensus] add test for non-voters in consensus queue Added an integration test to verify that the consensus queue correctly distinguishes between voter and non-voter replica acknowledgements of the Raft consensus messages. The test is currently disabled because it fails: currently, the consensus queue does not differentiate between Raft acknowledgement messages sent by voter and non-voter replicas. Change-Id: I16382b75256a32d695d8bc9ce020f40276114511 Reviewed-on: http://gerrit.cloudera.org:8080/8867 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy --- M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc 1 file changed, 108 insertions(+), 0 deletions(-) Approvals: Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8867 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I16382b75256a32d695d8bc9ce020f40276114511 Gerrit-Change-Number: 8867 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy
[kudu-CR] KUDU-2236: use debug logging where appropriate
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8961 to look at the new patch set (#2). Change subject: KUDU-2236: use debug logging where appropriate .. KUDU-2236: use debug logging where appropriate The previous fixes for KUDU-2236 adjusted the test to ensure its success when they should have instead addressed the regression in logging behavior. I.e. testCloseShortlyAfterOpen() should not catch any scary log messages because it itself isn't doing anything scary. This patch addresses the fact that exception messages are spewed even when we might expect them because a connection has been explicitly closed. In such cases, DEBUG-level messages will be logged instead of INFO- or WARN-level messages. CapturingLogAppenders can now specify what log levels to read from, and testCloseShortlyAfterOpen will now parse INFO-level messages or higher (i.e. not DEBUG-level). Change-Id: I780df2eeb51a14b65dd4283dfe9d4d6daf909661 --- M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/util/CapturingLogAppender.java 4 files changed, 30 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/8961/2 -- To view, visit http://gerrit.cloudera.org:8080/8961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I780df2eeb51a14b65dd4283dfe9d4d6daf909661 Gerrit-Change-Number: 8961 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-2236: add debug logging where appropriate
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8961 Change subject: KUDU-2236: add debug logging where appropriate .. KUDU-2236: add debug logging where appropriate The previous fixes for KUDU-2236 adjusted the test to ensure its success when they should have instead addressed the regression in logging behavior. I.e. testCloseShortlyAfterOpen() should not catch any scary log messages because it itself isn't doing anything scary. This patch addresses the fact that exception messages are spewed even when we might expect them because a connection has been explicitly closed. In such cases, DEBUG-level messages will be logged instead of INFO- or WARN-level messages. CapturingLogAppenders can now specify what log levels to read from, and testCloseShortlyAfterOpen will now parse INFO-level messages or higher (i.e. not DEBUG-level). Change-Id: I780df2eeb51a14b65dd4283dfe9d4d6daf909661 --- M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/Connection.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/util/CapturingLogAppender.java 4 files changed, 30 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/8961/1 -- To view, visit http://gerrit.cloudera.org:8080/8961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I780df2eeb51a14b65dd4283dfe9d4d6daf909661 Gerrit-Change-Number: 8961 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong
[kudu-CR] [java client] update propagated TS for AUTO FLUSH SYNC flush mode
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8837 ) Change subject: [java client] update propagated TS for AUTO_FLUSH_SYNC flush mode .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8837 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf0ca58b10842cb15ed5db7bcd4694c4d8cc3a89 Gerrit-Change-Number: 8837 Gerrit-PatchSet: 6 Gerrit-Owner: Hao Hao Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 19:03:35 + Gerrit-HasComments: No
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
David Ribeiro Alves has posted comments on this change. (
http://gerrit.cloudera.org:8080/8951 )
Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo
..
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/integration-tests/heavy-update-compaction-itest.cc
File src/kudu/integration-tests/heavy-update-compaction-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/integration-tests/heavy-update-compaction-itest.cc@206
PS6, Line 206: // Scan the updated rows and ensure the final values are
present.
: KuduScanner scanner(table_.get());
: ASSERT_OK(scanner.SetFaultTolerant());
: ASSERT_OK(scanner.AddConjunctPredicate(
: table_->NewComparisonPredicate(
: "key", KuduPredicate::LESS,
KuduValue::FromInt(FLAGS_rows;
:
: // Walking the updates can take a long time.
: scanner.SetTimeoutMillis(120 * 1000);
:
: LOG_TIMING(INFO, "scanning") {
: ASSERT_OK(scanner.Open());
: vector rows;
: size_t final_values_offset = 0;
: while (scanner.HasMoreRows()) {
: ASSERT_OK(scanner.NextBatch(&rows));
: for (const auto & row : rows) {
: for (int idx = 1; idx <= 5; idx++) {
: ASSERT_GT(final_values.size(), final_values_offset);
: Slice actual_val;
: ASSERT_OK(row.GetString(idx, &actual_val));
: EXPECT_EQ(actual_val,
final_values[final_values_offset++]);
: }
: }
: }
: }
Thanks for doing this. I'm surprised that it might take so long. I guess this
is one of the cases where skipping updates is going to make a huge difference.
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/tablet/rowset_info.h
File src/kudu/tablet/rowset_info.h:
http://gerrit.cloudera.org:8080/#/c/8951/6/src/kudu/tablet/rowset_info.h@21
PS6, Line 21:
extra line
--
To view, visit http://gerrit.cloudera.org:8080/8951
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb
Gerrit-Change-Number: 8951
Gerrit-PatchSet: 6
Gerrit-Owner: Dan Burkert
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: David Ribeiro Alves
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 08 Jan 2018 19:02:48 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2251: rowset size can overflow int in RowSetInfo
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8951 ) Change subject: KUDU-2251: rowset size can overflow int in RowSetInfo .. Patch Set 6: OK I think I've satisfied everyone's requests WRT tests, however it now appears that the IT is taking more than 120 seconds to scan 500 rows in TSAN builds... so I'll be looking into that. May just have to bump the timeout, but maybe I can figure out why it's taking that long. -- To view, visit http://gerrit.cloudera.org:8080/8951 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74975cdab605b51617d93d1ae98ef72ce87e35cb Gerrit-Change-Number: 8951 Gerrit-PatchSet: 6 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Jan 2018 18:59:36 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. Patch Set 5: Code-Review+2 Ah, my bad, gerrit's column mark at 76 chars tricked me. Thanks for addressing my nits. -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 5 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 18:57:07 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
David Ribeiro Alves has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader This commit prevents a possible race condition between getSplits() method and TableRecordReader in the KuduTableInputFormat, when both try to access and shutdown the KuduClient. Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. This increases the number of opened Kudu clients by a MR application at most by one (The one that was shared by getSplits() with a TableRecordReader) Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect in total. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Reviewed-on: http://gerrit.cloudera.org:8080/8921 Tested-by: Kudu Jenkins Reviewed-by: David Ribeiro Alves --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) Approvals: Kudu Jenkins: Verified David Ribeiro Alves: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 6 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] consensus: Fix NON VOTER ack-counting bug
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8868 ) Change subject: consensus: Fix NON_VOTER ack-counting bug .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8868 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13143e9bb4b76af3fd6dada28fcec05b27d24476 Gerrit-Change-Number: 8868 Gerrit-PatchSet: 6 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Mon, 08 Jan 2018 15:31:18 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Clemens Valiente has posted comments on this change. ( http://gerrit.cloudera.org:8080/8921 ) Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. Patch Set 4: > Patch Set 4: > > (2 comments) I added the clarification. The lines should below 80 cols already if I am not mistaken? -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 4 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 08 Jan 2018 10:39:59 + Gerrit-HasComments: No
[kudu-CR] KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader
Hello David Ribeiro Alves, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8921 to look at the new patch set (#5). Change subject: KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader .. KUDU-2249 Avoid sharing the client between the InputFormat and RecordReader This commit prevents a possible race condition between getSplits() method and TableRecordReader in the KuduTableInputFormat, when both try to access and shutdown the KuduClient. Both are sharing the same client and shut it down after use. In some scenarios the client might still be accessed after that and throwing an error. So the TableRecordReader gets its own client with this commit. This increases the number of opened Kudu clients by a MR application at most by one (The one that was shared by getSplits() with a TableRecordReader) Also clarified the behaviour of MR applications and how many open Kudu clients one might have to expect in total. Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 --- M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java 1 file changed, 45 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/8921/5 -- To view, visit http://gerrit.cloudera.org:8080/8921 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I24f45ee9253790c5348cabd0afe6c6a4b6d3f3d4 Gerrit-Change-Number: 8921 Gerrit-PatchSet: 5 Gerrit-Owner: Clemens Valiente Gerrit-Reviewer: Clemens Valiente Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins
