[1/3] kudu git commit: [tools] ksck improvements [7/n] Add JSON output option to ksck

[mpercy]

Repository: kudu
Updated Branches:
  refs/heads/master 72df429ec -> 05d594867


[tools] ksck improvements [7/n] Add JSON output option to ksck

This adds support for JSON output to ksck, using a new flag
--ksck_format that supports four options: 'plain_concise',
'plain_full', 'json_pretty', and 'json_concise'.
'plain_concise' formatting is the default: it's the original
formatting and it is not changed by this patch. 'plain_full'
supersedes the --verbose flag. The 'json_pretty' option
pretty-prints a json representation of all the information
gathered by ksck. 'json_compact' ugly-prints the same, and is
suitable for parsing by other programs, like jq.

There's a knock-on effect of how this change is implemented: since
KsckResults is translated to PB and then to JSON via generic PB-to-JSON
code, the printing for CONSENSUS_MISMATCH health has changed to
match its PB stringification. Previously it was stringified as
UNAVAILABLE. Note that it isn't necessarily true that
CONSENSUS_MISMATCH implies unavailable, anyway, since, e.g.
a replica not yet aware of a new leader will cause a
CONSENSUS_MISMATCH state on a tablet that's available.

Here's a sample of the 4 formats run against the same 1-table,
8-tablet cluster:

plain_concise: 
https://gist.github.com/wdberkeley/674a8b0322c4c0ad6e8eb4ef79664d37
plain_full: https://gist.github.com/wdberkeley/841c92cf4e500782b0dc3a30a8c1cbd8
json_pretty: https://gist.github.com/wdberkeley/04dca6dd5ec7a10ad10c4bd30decab35
json_compact: 
https://gist.github.com/wdberkeley/283d48ad248a26073a30e013e19443c3

Change-Id: Ib5da0752f8e41c022611253c300450368f6ae969
Reviewed-on: http://gerrit.cloudera.org:8080/10288
Reviewed-by: Alexey Serbin 
Reviewed-by: Andrew Wong 
Tested-by: Will Berkeley 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/ee0a75c5
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/ee0a75c5
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/ee0a75c5

Branch: refs/heads/master
Commit: ee0a75c5a22751241ff25c8f7b9c0a2654fbd767
Parents: 72df429
Author: Will Berkeley 
Authored: Tue May 1 13:17:57 2018 -0700
Committer: Will Berkeley 
Committed: Fri May 11 21:32:01 2018 +

--
 src/kudu/tools/CMakeLists.txt |   1 +
 src/kudu/tools/ksck-test.cc   | 490 ++---
 src/kudu/tools/ksck.cc|  35 ++-
 src/kudu/tools/ksck_results.cc| 227 -
 src/kudu/tools/ksck_results.h |  25 +-
 src/kudu/tools/tool.proto | 114 +++
 src/kudu/tools/tool_action_cluster.cc |   2 +-
 7 files changed, 819 insertions(+), 75 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/ee0a75c5/src/kudu/tools/CMakeLists.txt
--
diff --git a/src/kudu/tools/CMakeLists.txt b/src/kudu/tools/CMakeLists.txt
index 832bb1d..1a7a0fa 100644
--- a/src/kudu/tools/CMakeLists.txt
+++ b/src/kudu/tools/CMakeLists.txt
@@ -31,6 +31,7 @@ add_library(tool_proto
 target_link_libraries(tool_proto
   kudu_common_proto
   protobuf
+  tablet_proto
   wire_protocol_proto)
 
 ###

http://git-wip-us.apache.org/repos/asf/kudu/blob/ee0a75c5/src/kudu/tools/ksck-test.cc
--
diff --git a/src/kudu/tools/ksck-test.cc b/src/kudu/tools/ksck-test.cc
index 18d28ef..18db2ce 100644
--- a/src/kudu/tools/ksck-test.cc
+++ b/src/kudu/tools/ksck-test.cc
@@ -15,8 +15,12 @@
 // specific language governing permissions and limitations
 // under the License.
 
+#include "kudu/tools/ksck.h"
+
+#include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -28,6 +32,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "kudu/common/schema.h"
 #include "kudu/consensus/metadata.pb.h"
@@ -36,8 +41,8 @@
 #include "kudu/gutil/strings/substitute.h"
 #include "kudu/tablet/metadata.pb.h"
 #include "kudu/tablet/tablet.pb.h"
-#include "kudu/tools/ksck.h"
 #include "kudu/tools/ksck_results.h"
+#include "kudu/util/jsonreader.h"
 #include "kudu/util/scoped_cleanup.h"
 #include "kudu/util/status.h"
 #include "kudu/util/test_macros.h"
@@ -49,6 +54,7 @@ DECLARE_string(color);
 namespace kudu {
 namespace tools {
 
+using std::ostringstream;
 using std::shared_ptr;
 using std::static_pointer_cast;
 using std::string;
@@ -336,6 +342,7 @@ class KsckTest : public KuduTest {
 pb.set_tablet_id(tablet_id);
 pb.set_table_name("fake-table");
 pb.set_state(is_running ? tablet::RUNNING : tablet::FAILED);
+pb.set_tablet_data_state(TabletDataState::TABLET_DATA_UNKNOWN);
 InsertOrDie(>tablet_status_map_, tablet_id, pb);
   }
 
@@ -346,6 +353,12 @@ class 

[2/3] kudu git commit: A new Jepsen checker for READ_YOUR_WRITES scan mode

[mpercy]

A new Jepsen checker for READ_YOUR_WRITES scan mode

This patch adds a new Jepsen checker to validate READ_YOUR_WRITES scan
mode. In this validation mode, each Jepsen client writes unique values
in a shared table concurrently. The checker generates two kind of
operations: 'count' which counts the number of rows of the table, 'add'
which inserts a unique value to the table. It validates that the total
row count of the table read by the client is always greater than or
equal to the count of previous writes by the same client, and the row
count never go down from the previous reads.

Change-Id: I92d5c0e3b91af58576eb6cd408d922ec7c0fef6c
Reviewed-on: http://gerrit.cloudera.org:8080/9526
Reviewed-by: David Ribeiro Alves 
Tested-by: Hao Hao 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/ba0a8126
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/ba0a8126
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/ba0a8126

Branch: refs/heads/master
Commit: ba0a8126fb95778a9964864b55fc189441393052
Parents: ee0a75c
Author: hahao 
Authored: Mon Mar 26 12:17:55 2018 -0700
Committer: Hao Hao 
Committed: Fri May 11 21:35:51 2018 +

--
 .../src/main/clojure/jepsen/kudu/sets.clj   | 217 +++
 .../src/main/clojure/jepsen/kudu/table.clj  |  10 +
 .../src/test/clojure/jepsen/kudu_test.clj   |  10 +-
 3 files changed, 234 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/ba0a8126/java/kudu-jepsen/src/main/clojure/jepsen/kudu/sets.clj
--
diff --git a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/sets.clj 
b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/sets.clj
new file mode 100644
index 000..9bfdcd6
--- /dev/null
+++ b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/sets.clj
@@ -0,0 +1,217 @@
+;; Licensed to the Apache Software Foundation (ASF) under one
+;; or more contributor license agreements. See the NOTICE file
+;; distributed with this work for additional information
+;; regarding copyright ownership. The ASF licenses this file
+;; to you under the Apache License, Version 2.0 (the
+;; "License"); you may not use this file except in compliance
+;; with the License. You may obtain a copy of the License at
+;;
+;;   http://www.apache.org/licenses/LICENSE-2.0
+;;
+;; Unless required by applicable law or agreed to in writing,
+;; software distributed under the License is distributed on an
+;; "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+;; KIND, either express or implied. See the License for the
+;; specific language governing permissions and limitations
+;; under the License.
+
+;; The 'sets' checker validates that:
+;; 1) the total row count of a shared table read by a client is
+;;greater than or equal to the count of successful writes
+;;performed by that client.
+;; 2) the row count never goes down from the previous reads of
+;;the same client.
+;;
+;; For example, sets' checker considers a history as follows valid:
+;;
+;;   [{:type :invoke, :f :add, :value 1, :process 0}
+;;{:type :ok, :f :add, :value 1, :process 0}
+;;{:type :invoke, :f :count, :value nil, :process 1}
+;;{:type :ok, :f :count, :value 2, :process 1}
+;;{:type :invoke, :f :add, :value 22, :process 0}
+;;{:type :ok, :f :add, :value 22, :process 0}
+;;{:type :invoke, :f :count, :value nil, :process 0}
+;;{:type :ok, :f :count, :value 2, :process 0}]
+;;
+;; while the following sequence as invalid. Since after process 0
+;; successfully inserted three rows, the total number of rows
+;; returned from the count operation of the same process is two,
+;; which conflicts with the validation constraints:
+;;
+;;   [{:type :invoke, :f :add, :value 1, :process 0}
+;;{:type :ok, :f :add, :value 1, :process 0}
+;;{:type :invoke, :f :add, :value 2, :process 0}
+;;{:type :ok, :f :add, :value 2, :process 0}
+;;{:type :invoke, :f :count, :value nil, :process 0}
+;;{:type :ok, :f :count, :value 2, :process 0}
+;;{:type :invoke, :f :count, :value nil, :process 1}
+;;{:type :ok, :f :count, :value 2, :process 1}
+;;{:type :invoke, :f :add, :value 13, :process 1}
+;:{:type :ok, :f :add, :value 13, :process 1}
+;;{:type :invoke, :f :add, :value 22, :process 0}
+;;{:type :ok, :f :add, :value 22, :process 0}
+;;{:type :invoke, :f :count, :value nil, :process 0}
+;;{:type :ok, :f :count, :value 2, :process 0}
+;;{:type :invoke, :f :add, :value 250, :process 1}
+;;{:type :ok, :f :add, :value 250, :process 1}]
+;;
+;; This checker model is a variation of cockroach 'sets' checker,
+;; see 

[3/3] kudu git commit: [docs] Add a bit of information about tombstoned replicas

[mpercy]

[docs] Add a bit of information about tombstoned replicas

This adds a quick paragraph to the troubleshooting docs about tombstoned
replicas. I've seen quite a few users be alarmed by tombstones and
believe they need to be dealt with somehow, possibly by deletion. Of
course, this is dangerous, and having docs to point people towards will
be nice, and may prevent some user consternation.

Change-Id: I876aeb1ffeb7251fe36fe6022b722ed88997cdbe
Reviewed-on: http://gerrit.cloudera.org:8080/10372
Tested-by: Will Berkeley 
Reviewed-by: Mike Percy 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/05d59486
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/05d59486
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/05d59486

Branch: refs/heads/master
Commit: 05d594867c6a442691274b9b9a402b804eca1516
Parents: ba0a812
Author: Will Berkeley 
Authored: Fri May 11 00:58:57 2018 -0700
Committer: Mike Percy 
Committed: Fri May 11 23:25:27 2018 +

--
 docs/troubleshooting.adoc | 16 
 1 file changed, 16 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/05d59486/docs/troubleshooting.adoc
--
diff --git a/docs/troubleshooting.adoc b/docs/troubleshooting.adoc
index edb8991..826b2b8 100644
--- a/docs/troubleshooting.adoc
+++ b/docs/troubleshooting.adoc
@@ -602,3 +602,19 @@ link:http://kudu.apache.org/releases/[releases page] has 
links to documentation
 for previous versions of Kudu; for the latest release, see the
 link:known_issues.html[known issues page].
 
+[[tombstoned_or_stopped_tablets]]
+=== Tombstoned or STOPPED tablet replicas
+
+Users may notice some replicas on a tablet server are in a STOPPED state, and
+remain on the server indefinitely. These replicas are tombstones. A tombstone
+indicates that the tablet server once held a bona fide replica of its tablet.
+For example, if a tablet server goes down and its replicas are re-replicated
+elsewhere, if the tablet server rejoins the cluster its replicas will become
+tombstones. A tombstone will remain until the table it belongs to is deleted, 
or
+a new replica of the same tablet is placed on the tablet server. A count of
+tombstoned replicas and details of each one are available on the /tablets page
+of the tablet server web UI.
+
+The Raft consensus algorithm that Kudu uses for replication requires tombstones
+for correctness in certain rare situations. They consume minimal resources and
+hold no data. They must not be deleted.

kudu git commit: [docs] Update macOS build docs

[danburkert]

Repository: kudu
Updated Branches:
  refs/heads/master c18e7e1a6 -> 72df429ec


[docs] Update macOS build docs

* tapping homebrew/dupes is unnecessary now as it's now deprecated and
all formulae have been migrated to homebrew-core or deleted

Change-Id: Ic2a8aa7f58b7401115f36f42634d80b4d64e9794
Reviewed-on: http://gerrit.cloudera.org:8080/10356
Reviewed-by: Dan Burkert 
Tested-by: Dan Burkert 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/72df429e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/72df429e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/72df429e

Branch: refs/heads/master
Commit: 72df429ecf95eaa01dee4743290395b650ec2ed4
Parents: c18e7e1
Author: Attila Bukor 
Authored: Fri May 11 18:16:29 2018 +0200
Committer: Dan Burkert 
Committed: Fri May 11 17:04:20 2018 +

--
 docs/installation.adoc | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/72df429e/docs/installation.adoc
--
diff --git a/docs/installation.adoc b/docs/installation.adoc
index 8d22655..236c325 100644
--- a/docs/installation.adoc
+++ b/docs/installation.adoc
@@ -551,7 +551,6 @@ for more information.
 . Install the prerequisite libraries, if they are not installed.
 +
 
-$ brew tap homebrew/dupes
 $ brew install autoconf automake cmake git krb5 libtool openssl pkg-config 
pstree