kudu git commit: KUDU-2293 fix cleanup after failed copies

2018-04-10 Thread awong 
Repository: kudu
Updated Branches:
  refs/heads/master 745ebab2f -> 03d1fcb14


KUDU-2293 fix cleanup after failed copies

Before, when a tablet server failed to tablet copy, Kudu would perform a
best-effort cleanup of the partially-copied replica and leave the tablet
tombstoned. If this tombstoning were to fail due to disk issues (e.g.
out of space), Kudu would allow this and tablet would remain in
TABLET_DATA_COPYING both in-memory and on-disk. This would lead to a
FATAL error if another tablet copy were started for the same replica, as
the server would attempt to copy over a replica with data already marked
TABLET_DATA_COPYING.

This behavior arose from trying to balance two invariants:
- keep on-disk state consistent with in-memory state when possible
- when a tablet copy fails, leave it in as dead of a state as we can
  (i.e. TABLET_DATA_TOMBSTONED with no transitions in progress)

This patch updates the Abort() logic to lean towards the latter
invariant: if a tablet copy fails, at least its in-memory state will be
set as TABLET_DATA_TOMBSTONED. This may not be true on-disk, but that's
okay because either 1) the tablet server will eventually overwrite it
via another tablet copy (at which time its data must _not_ be in the
TABLET_DATA_COPYING state), or 2) the server will be restarted and the
tablet will be tombstoned upon seeing a non-TABLET_DATA_READY state
anyway.

We use the tablet copy internal state machine to determine whether to do
anything during Abort(). This wasn't always right before, since the
state machine didn't accurately reflect when cleanup was necessary (e.g.
the copy would be set to kStarted only at the end of Start(), so if
Abort() were called due to a failure in Start(), no cleanup was done).
This patch updates the state machine to account for this by introducing a
new state kStarting that indicates that there exists state to clean up
even if Start() has not completed, and by moving the setting of
kFinished such that cleanup is done even if Finish() fails.

A test is added to tablet_copy-itest that tests failures to copy,
ensuring that the tablet is left in such an state that further copies
are possible without crashing. The test uses EIO injection to fail the
copies, but the logic is the same as if full disks were used instead.
Another is added to tablet_copy_client-test that tests getting into the
various states in the updated tablet copy state machine.

Change-Id: I0459d484a3064aa2de392328e2910c9f6741be04
Reviewed-on: http://gerrit.cloudera.org:8080/9452
Reviewed-by: Mike Percy 
Tested-by: Andrew Wong 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/03d1fcb1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/03d1fcb1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/03d1fcb1

Branch: refs/heads/master
Commit: 03d1fcb1467e4e29ead191d875a53e490f1d1372
Parents: 745ebab
Author: Andrew Wong 
Authored: Fri Feb 23 14:58:58 2018 -0800
Committer: Andrew Wong 
Committed: Wed Apr 11 02:46:31 2018 +

--
 src/kudu/integration-tests/tablet_copy-itest.cc |  92 
 src/kudu/tablet/tablet_metadata.cc  |  15 +--
 src/kudu/tablet/tablet_metadata.h   |  16 +--
 src/kudu/tserver/tablet_copy_client-test.cc | 104 +--
 src/kudu/tserver/tablet_copy_client.cc  |  60 ---
 src/kudu/tserver/tablet_copy_client.h   |  27 -
 6 files changed, 272 insertions(+), 42 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/03d1fcb1/src/kudu/integration-tests/tablet_copy-itest.cc
--
diff --git a/src/kudu/integration-tests/tablet_copy-itest.cc 
b/src/kudu/integration-tests/tablet_copy-itest.cc
index e52bead..e8fc265 100644
--- a/src/kudu/integration-tests/tablet_copy-itest.cc
+++ b/src/kudu/integration-tests/tablet_copy-itest.cc
@@ -95,6 +95,7 @@ DEFINE_int32(test_delete_leader_num_writer_threads, 1,
 using kudu::client::KuduSchema;
 using kudu::client::KuduSchemaFromSchema;
 using kudu::client::KuduTableCreator;
+using kudu::cluster::ExternalMiniClusterOptions;
 using kudu::cluster::ExternalTabletServer;
 using kudu::consensus::COMMITTED_OPID;
 using kudu::consensus::ConsensusMetadataManager;
@@ -379,6 +380,97 @@ TEST_F(TabletCopyITest, TestListTabletsDuringTabletCopy) {
   NO_FATALS(cluster_->AssertNoCrashes());
 }
 
+// Regression test for KUDU-2293.
+TEST_F(TabletCopyITest, TestCopyAfterFailedCopy) {
+  MonoDelta kTimeout = MonoDelta::FromSeconds(30);
+  const int kTsIndex = 1; // Pick an arbitrary tserver to observe.
+  // Use 2 data dirs so we can fail a directory without crashing the server.
+  ExternalMiniClusterOptions cluster_opts;
+

kudu git commit: [jepsen] update pattern for kudu-master-see-tservers

2018-04-10 Thread alexey 
Repository: kudu
Updated Branches:
  refs/heads/master 05bec2c1c -> 745ebab2f


[jepsen] update pattern for kudu-master-see-tservers

Changelist 91dd090 introduced a small change in the output of
'kudu cluster ksck' which the kudu-master-see-tservers used to
check whether the output of the tool matches the expected pattern.

Change-Id: Ie26990db395440a52bc84fce6d704883199e9427
Reviewed-on: http://gerrit.cloudera.org:8080/9978
Reviewed-by: Adar Dembo 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/745ebab2
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/745ebab2
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/745ebab2

Branch: refs/heads/master
Commit: 745ebab2f4fed88af72bce0c6e1b8a369abd4779
Parents: 05bec2c
Author: Alexey Serbin 
Authored: Tue Apr 10 14:58:21 2018 -0700
Committer: Alexey Serbin 
Committed: Tue Apr 10 22:37:54 2018 +

--
 java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/745ebab2/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
--
diff --git a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj 
b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
index 2fa75a2..ad41b8f 100644
--- a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
+++ b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj
@@ -133,7 +133,7 @@
   "Whether the Kudu master sees the specified number of tablet servers."
   [test node tservers-count]
   (let [pattern (str "Fetched info from all "
- (str tservers-count)" Tablet Servers")]
+ (str tservers-count)" tablet servers")]
 (try (c/exec :sudo :-u kudu-uname (kudu-cli test node) :cluster :ksck node 
|
  :grep pattern)
  true

[3/3] kudu git commit: user: fix a possible integer overflow

2018-04-10 Thread adar 
user: fix a possible integer overflow

This was flagged by a run of Fortify, though I think it's pretty dubious
given the sysconf() contract, which returns either -1 or a positive number.

Change-Id: I08b4e47862b0f05558c4420c9b5d6ddd53ccd156
Reviewed-on: http://gerrit.cloudera.org:8080/9975
Reviewed-by: Dan Burkert 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/05bec2c1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/05bec2c1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/05bec2c1

Branch: refs/heads/master
Commit: 05bec2c1c0740ad6319d65cb5e4d5507e9b6c818
Parents: d18be56
Author: Adar Dembo 
Authored: Tue Apr 10 11:03:52 2018 -0700
Committer: Adar Dembo 
Committed: Tue Apr 10 20:52:46 2018 +

--
 src/kudu/util/user.cc | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/05bec2c1/src/kudu/util/user.cc
--
diff --git a/src/kudu/util/user.cc b/src/kudu/util/user.cc
index 2d12267..f44e040 100644
--- a/src/kudu/util/user.cc
+++ b/src/kudu/util/user.cc
@@ -21,6 +21,7 @@
 #include 
 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -44,10 +45,10 @@ Status DoGetLoggedInUser(string* user_name) {
   struct passwd pwd;
   struct passwd *result;
 
-  size_t bufsize = sysconf(_SC_GETPW_R_SIZE_MAX);
-  if (bufsize == -1) {  // Value was indeterminate.
-bufsize = 16384;// Should be more than enough, per the man page.
-  }
+  // Get the system-defined limit for usernames. If the value was 
indeterminate,
+  // use a constant that should be more than enough, per the man page.
+  int64_t retval = sysconf(_SC_GETPW_R_SIZE_MAX);
+  size_t bufsize = retval > 0 ? retval : 16384;
 
   gscoped_ptr buf(static_cast(malloc(bufsize)));
   if (buf.get() == nullptr) {

[1/3] kudu git commit: [tools] ksck improvements [3/n]: master consensus checks

2018-04-10 Thread adar 
Repository: kudu
Updated Branches:
  refs/heads/master 00c2754ca -> 05bec2c1c


[tools] ksck improvements [3/n]: master consensus checks

This patch adds consensus consistency checks and a consensus matrix
for the master tablet. It's a little trickier than for tablets,
because there's no uuid available for an unavailable master.

Here's a sample matrix when a master is down:
WARNING: masters have consensus conflicts  All reported masters are:
  A = c9afe67944784c70ab13989354c10f9e
  B =  (localhost:7052)
  C = 488cf2ab551b422aa8c683a054cdcde3
  D = 62dc9b532b084a459130f1676d8e1998
 Config source |Replicas| Current term | Config index | 
Committed?
---++--+--+
 A | A   C*  D  | 4| -1   | Yes
 B | [config not available] |  |  |
 C | A   C*  D  | 4| -1   | Yes

Change-Id: I1015854759debb3f1acf4bf9eb143260547f4a4b
Reviewed-on: http://gerrit.cloudera.org:8080/9883
Reviewed-by: Attila Bukor 
Tested-by: Will Berkeley 
Reviewed-by: Andrew Wong 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/70221010
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/70221010
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/70221010

Branch: refs/heads/master
Commit: 702210106d42433a9c255f94b488f8d4bdd9986a
Parents: 00c2754
Author: Will Berkeley 
Authored: Sat Mar 31 14:36:29 2018 -0700
Committer: Will Berkeley 
Committed: Tue Apr 10 19:46:29 2018 +

--
 src/kudu/integration-tests/cluster_verifier.cc |   6 +-
 src/kudu/tools/CMakeLists.txt  |   1 +
 src/kudu/tools/ksck-test.cc|  92 +++-
 src/kudu/tools/ksck.cc | 247 +---
 src/kudu/tools/ksck.h  |  27 ++-
 src/kudu/tools/ksck_remote-test.cc |   7 +
 src/kudu/tools/ksck_remote.cc  |  26 ++-
 src/kudu/tools/tool_action_cluster.cc  |   2 +
 8 files changed, 306 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/70221010/src/kudu/integration-tests/cluster_verifier.cc
--
diff --git a/src/kudu/integration-tests/cluster_verifier.cc 
b/src/kudu/integration-tests/cluster_verifier.cc
index 5332404..407df06 100644
--- a/src/kudu/integration-tests/cluster_verifier.cc
+++ b/src/kudu/integration-tests/cluster_verifier.cc
@@ -109,9 +109,11 @@ Status ClusterVerifier::RunKsck() {
   // we shouldn't consider fatal.
   ksck->set_check_replica_count(false);
 
-  // The first two calls do not depend on each other, though their results are
-  // correlated. The subsequent calls depend on CheckClusterRunning().
+  // The CheckMaster* calls are independent of CheckClusterRunning, though
+  // their results are correlated. The subsequent calls depend on
+  // CheckClusterRunning().
   RETURN_NOT_OK(ksck->CheckMasterHealth());
+  RETURN_NOT_OK(ksck->CheckMasterConsensus());
   RETURN_NOT_OK(ksck->CheckClusterRunning());
   RETURN_NOT_OK(ksck->FetchTableAndTabletInfo());
   RETURN_NOT_OK(ksck->FetchInfoFromTabletServers());

http://git-wip-us.apache.org/repos/asf/kudu/blob/70221010/src/kudu/tools/CMakeLists.txt
--
diff --git a/src/kudu/tools/CMakeLists.txt b/src/kudu/tools/CMakeLists.txt
index 06fe484..0135994 100644
--- a/src/kudu/tools/CMakeLists.txt
+++ b/src/kudu/tools/CMakeLists.txt
@@ -72,6 +72,7 @@ add_library(ksck
 target_link_libraries(ksck
   consensus
   kudu_tools_util
+  master
   master_proto
   server_base_proto
   tserver_proto

http://git-wip-us.apache.org/repos/asf/kudu/blob/70221010/src/kudu/tools/ksck-test.cc
--
diff --git a/src/kudu/tools/ksck-test.cc b/src/kudu/tools/ksck-test.cc
index 530bc0b..aeeff2d 100644
--- a/src/kudu/tools/ksck-test.cc
+++ b/src/kudu/tools/ksck-test.cc
@@ -24,6 +24,7 @@
 #include 
 #include 
 
+#include 
 #include 
 #include 
 #include 
@@ -77,11 +78,14 @@ class MockKsckMaster : public KsckMaster {
   }
 
   Status FetchConsensusState() override {
-return Status::OK();
+return fetch_cstate_status_;
   }
 
-  // Public because the unit tests mutate this variable directly.
+  // Public because the unit tests mutate these variables directly.
   Status fetch_info_status_;
+  Status fetch_cstate_status_;
+  using KsckMaster::uuid_;
+  using KsckMaster::cstate_;
 };
 
 class MockKsckTabletServer : public KsckTabletServer {
@@ -161,11 +165,25 @@ class

[2/3] kudu git commit: [tools] ksck improvements [4/n]: KUDU-2271 and KUDU-2310

2018-04-10 Thread adar 
[tools] ksck improvements [4/n]: KUDU-2271 and KUDU-2310

This improves formatting in ksck output and ensures that the tablet ids
appear in the output in verbose mode (KUDU-2271).

It also revamps how results are printed when tablet id filters are
specified, omitting conclusions about entire tables. It's now clear from
ksck's messages whether it is checking individual tablets using
tablet filters or checking whole tables. This addresses KUDU-2310 as
well, since it makes it clear what sort of filters are being applied.
The summary table is unchanged since it's nice whether a table or tablet
filter is applied.

Change-Id: I2371b704a7c2ebb1b5a4822e3b4d852443c200c5
Reviewed-on: http://gerrit.cloudera.org:8080/9912
Reviewed-by: Attila Bukor 
Tested-by: Will Berkeley 
Reviewed-by: Andrew Wong 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d18be566
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d18be566
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d18be566

Branch: refs/heads/master
Commit: d18be566aeda331bb7e28fe14f336e2ae95c3a53
Parents: 7022101
Author: Will Berkeley 
Authored: Mon Apr 2 02:38:25 2018 -0700
Committer: Will Berkeley 
Committed: Tue Apr 10 20:18:03 2018 +

--
 src/kudu/tools/ksck-test.cc | 52 ++---
 src/kudu/tools/ksck.cc  | 71 +++-
 src/kudu/tools/ksck.h   |  4 +++
 3 files changed, 107 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/d18be566/src/kudu/tools/ksck-test.cc
--
diff --git a/src/kudu/tools/ksck-test.cc b/src/kudu/tools/ksck-test.cc
index aeeff2d..5d9f9e8 100644
--- a/src/kudu/tools/ksck-test.cc
+++ b/src/kudu/tools/ksck-test.cc
@@ -231,16 +231,17 @@ class KsckTest : public KuduTest {
 table->set_tablets({ tablet });
   }
 
-  void CreateOneSmallReplicatedTable() {
+  void CreateOneSmallReplicatedTable(const string& table_name = "test",
+ const string& tablet_id_prefix = "") {
 int num_replicas = 3;
 int num_tablets = 3;
 CreateDefaultAssignmentPlan(num_replicas * num_tablets);
-auto table = CreateAndAddTable("test", num_replicas);
+auto table = CreateAndAddTable(table_name, num_replicas);
 
 vector tablets;
 for (int i = 0; i < num_tablets; i++) {
   shared_ptr tablet(new KsckTablet(
-  table.get(), Substitute("tablet-id-$0", i)));
+  table.get(), Substitute("$0tablet-id-$1", tablet_id_prefix, i)));
   CreateAndFillTablet(tablet, num_replicas, true, true);
   tablets.push_back(tablet);
 }
@@ -276,8 +277,7 @@ class KsckTest : public KuduTest {
 
   shared_ptr CreateAndAddTable(const string& name, int 
num_replicas) {
 shared_ptr table(new KsckTable(name, Schema(), num_replicas));
-vector tables = { table };
-cluster_->tables_.assign(tables.begin(), tables.end());
+cluster_->tables_.push_back(table);
 return table;
   }
 
@@ -799,5 +799,47 @@ TEST_F(KsckTest, 
TestMasterNotReportingTabletServerWithConsensusConflict) {
   
/*unavailable_tables=*/ 0));
 }
 
+TEST_F(KsckTest, TestTableFiltersNoMatch) {
+  CreateOneSmallReplicatedTable();
+
+  ksck_->set_table_filters({ "fake-table" });
+  Status s = RunKsck();
+
+  // Every table we checked was healthy ;).
+  ASSERT_OK(s);
+  ASSERT_STR_CONTAINS(err_stream_.str(), "The cluster doesn't have any 
matching tables");
+}
+
+TEST_F(KsckTest, TestTableFilters) {
+  CreateOneSmallReplicatedTable();
+  CreateOneSmallReplicatedTable("other", "other-");
+
+  ksck_->set_table_filters({ "test" });
+  Status s = RunKsck();
+
+  ASSERT_OK(s);
+  ASSERT_STR_CONTAINS(err_stream_.str(), "The metadata for 1 table(s) is 
HEALTHY");
+}
+
+TEST_F(KsckTest, TestTabletFiltersNoMatch) {
+  CreateOneSmallReplicatedTable();
+
+  ksck_->set_tablet_id_filters({ "tablet-id-fake" });
+  Status s = RunKsck();
+
+  ASSERT_OK(s);
+  ASSERT_STR_CONTAINS(err_stream_.str(), "The cluster doesn't have any 
matching tablets");
+}
+
+TEST_F(KsckTest, TestTabletFilters) {
+  CreateOneSmallReplicatedTable();
+
+  ksck_->set_tablet_id_filters({ "tablet-id-0", "tablet-id-1" });
+  Status s = RunKsck();
+
+  ASSERT_OK(s);
+  ASSERT_STR_CONTAINS(err_stream_.str(), "The metadata for 2 tablet(s) is 
HEALTHY");
+}
+
 } // namespace tools
 } // namespace kudu

http://git-wip-us.apache.org/repos/asf/kudu/blob/d18be566/src/kudu/tools/ksck.cc
--
diff --git a/src/kudu/tools/ksck.cc b/src/kudu/tools/ksck.cc
index

[2/2] kudu git commit: KUDU-2191 (9/n): HMS Catalog should short-circuit no-op alter and create table calls

2018-04-10 Thread danburkert 
KUDU-2191 (9/n): HMS Catalog should short-circuit no-op alter and create table 
calls

This tweaks the HMS catalog class introduced in part 7 to be more
flexible with how tables are created and altered.  In particular, when
altering, renaming, or creating a table entry in the HMS, it returns an
OK status if the HMS already has the proper table entry, effectively
short-circuiting the operation. Some short-circuiting existed before,
but this makes it apply more uniformly in more situations.  This is
necessary for the notification listener to function correctly, since
it's not always possible to determine which notification log events have
aready been applied.

I also took the opportunity to refactor HMS catalog a bit, since
otherwise it would have been very unweildy to implement this
functionality. I've added some specific test cases to hms_catalog-test,
and some error message strings have changed slightly.

Change-Id: I9c3594db0db253bc88e032b8c5aa2c8667c49853
Reviewed-on: http://gerrit.cloudera.org:8080/9965
Reviewed-by: Adar Dembo 
Reviewed-by: Hao Hao 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/00c2754c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/00c2754c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/00c2754c

Branch: refs/heads/master
Commit: 00c2754cae8bd58bf3fa61ea95da74eed4133149
Parents: 21f651d
Author: Dan Burkert 
Authored: Mon Apr 9 15:46:47 2018 -0700
Committer: Dan Burkert 
Committed: Tue Apr 10 18:55:45 2018 +

--
 src/kudu/hms/hms_catalog-test.cc   |  68 ++-
 src/kudu/hms/hms_catalog.cc| 123 
 src/kudu/hms/hms_catalog.h |  22 +++-
 src/kudu/hms/hms_client.cc |   1 +
 src/kudu/hms/hms_client.h  |   1 +
 src/kudu/integration-tests/master_hms-itest.cc |   4 +-
 6 files changed, 134 insertions(+), 85 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/00c2754c/src/kudu/hms/hms_catalog-test.cc
--
diff --git a/src/kudu/hms/hms_catalog-test.cc b/src/kudu/hms/hms_catalog-test.cc
index d060320..af0244b 100644
--- a/src/kudu/hms/hms_catalog-test.cc
+++ b/src/kudu/hms/hms_catalog-test.cc
@@ -258,6 +258,17 @@ TEST_P(HmsCatalogTestParameterized, TestTableLifecycle) {
   ASSERT_OK(hms_catalog_->CreateTable(kTableId, kTableName, schema));
   NO_FATALS(CheckTable(kHmsDatabase, kHmsTableName, kTableId, schema));
 
+  // Create the table again. This should succeed since the table ID matches. 
The
+  // HMS catalog will automatically short-circuit creating the table.
+  // TODO(dan): once we have HMS catalog stats, assert that the op short 
circuits.
+  ASSERT_OK(hms_catalog_->CreateTable(kTableId, kTableName, schema));
+  NO_FATALS(CheckTable(kHmsDatabase, kHmsTableName, kTableId, schema));
+
+  // Create the table again, but with a different table ID.
+  Status s = hms_catalog_->CreateTable("new-table-id", kTableName, schema);
+  ASSERT_TRUE(s.IsAlreadyPresent()) << s.ToString();
+  NO_FATALS(CheckTable(kHmsDatabase, kHmsTableName, kTableId, schema));
+
   // Alter the table.
   SchemaBuilder b(schema);
   b.AddColumn("new_column", DataType::INT32);
@@ -305,70 +316,73 @@ TEST_F(HmsCatalogTest, TestLegacyTables) {
 // belong to external tables from other systems.
 TEST_F(HmsCatalogTest, TestExternalTable) {
   const string kTableId = "table-id";
-  const string kHmsDatabase = "default";
-
-  const string kHmsExternalTable = "external_table";
-  const string kExternalTableName = Substitute("$0.$1", kHmsDatabase, 
kHmsExternalTable);
-
-  const string kHmsKuduTable = "kudu_table";
-  const string kKuduTableName = Substitute("$0.$1", kHmsDatabase, 
kHmsKuduTable);
 
-  // Create the external table.
+  // Create the external table (default.ext).
   hive::Table external_table;
-  external_table.dbName = kHmsDatabase;
-  external_table.tableName = kHmsExternalTable;
+  external_table.dbName = "default";
+  external_table.tableName = "ext";
   external_table.tableType = HmsClient::kManagedTable;
   ASSERT_OK(hms_client_->CreateTable(external_table));
   // Retrieve the full HMS table object so it can be compared later (the HMS
   // fills in some fields during creation).
-  ASSERT_OK(hms_client_->GetTable(kHmsDatabase, kHmsExternalTable, 
_table));
+  ASSERT_OK(hms_client_->GetTable("default", "ext", _table));
 
   auto CheckExternalTable = [&] {
 hive::Table current_table;
-ASSERT_OK(hms_client_->GetTable(kHmsDatabase, kHmsExternalTable, 
_table));
+ASSERT_OK(hms_client_->GetTable("default", "ext", _table));
 ASSERT_EQ(current_table, external_table);
   };
 
-

[1/2] kudu git commit: Add GetFlags endpoint and tool

2018-04-10 Thread danburkert 
Repository: kudu
Updated Branches:
  refs/heads/master 337a731c4 -> 00c2754ca


 Add GetFlags endpoint and tool

This adds an rpc endpoint that retrieves gflags from servers.

It also includes a tool for retrieving flag values from servers.
By default, it returns only flags with non-default values. It
supports returning all flags, and also filtering flags by tags.

Example output from the tool run against a local cluster's master:

  flag  | value | 
default? |  tags
+---+--+-
 log_dir| /tmp/kudu/logs/master/0   | false 
   | stable
 heap_profile_path  | /tmp/kudu-master.56285| false 
   | advanced,stable
 log_filename   | kudu-master   | false 
   | stable
 webserver_interface| 127.0.0.1 | false 
   | advanced
 master_addresses   | 127.0.0.1:7053,127.0.0.1:7052,127.0.0.1:7051, | false 
   | stable
 fs_data_dirs   | /tmp/kudu/data/master/0/0 | false 
   | stable
 rpc_bind_addresses | 127.0.0.1:7051| false 
   | stable
 fs_wal_dir | /tmp/kudu/wal/master/0| false 
   | stable
 evict_failed_followers | false | false 
   | advanced
 webserver_port | 8051  | false 
   | stable

The rpc endpoint will also be used by ksck in a follow-up.

Change-Id: Ia35b4261099c1a3c6e2ff68e907c84df9a7ff699
Reviewed-on: http://gerrit.cloudera.org:8080/9948
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/21f651db
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/21f651db
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/21f651db

Branch: refs/heads/master
Commit: 21f651db1c51d80c9ce2761e3d07aa1129d1faca
Parents: 337a731
Author: Will Berkeley 
Authored: Mon Apr 2 16:13:21 2018 -0700
Committer: Will Berkeley 
Committed: Tue Apr 10 18:44:49 2018 +

--
 src/kudu/server/generic_service.cc | 36 ++-
 src/kudu/server/generic_service.h  |  6 +++
 src/kudu/server/server_base.proto  | 24 ++
 src/kudu/tools/kudu-tool-test.cc   | 48 
 src/kudu/tools/tool_action_common.cc   | 44 +++
 src/kudu/tools/tool_action_common.h|  5 +++
 src/kudu/tools/tool_action_master.cc   | 14 ++
 src/kudu/tools/tool_action_tserver.cc  | 14 ++
 src/kudu/tserver/tablet_server-test.cc | 68 +
 src/kudu/util/flags.cc | 26 +--
 src/kudu/util/flags.h  |  2 +
 11 files changed, 273 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/21f651db/src/kudu/server/generic_service.cc
--
diff --git a/src/kudu/server/generic_service.cc 
b/src/kudu/server/generic_service.cc
index e786c63..300e37f 100644
--- a/src/kudu/server/generic_service.cc
+++ b/src/kudu/server/generic_service.cc
@@ -17,9 +17,10 @@
 
 #include "kudu/server/generic_service.h"
 
-#include 
 #include 
+#include 
 #include 
+#include 
 
 #include 
 #include 
@@ -39,6 +40,7 @@
 #include "kudu/util/debug-util.h"
 #include "kudu/util/debug/leak_annotations.h" // IWYU pragma: keep
 #include "kudu/util/flag_tags.h"
+#include "kudu/util/flags.h"
 #include "kudu/util/status.h"
 
 DECLARE_string(time_source);
@@ -71,6 +73,38 @@ bool GenericServiceImpl::AuthorizeClient(const 
google::protobuf::Message* /*req*
 }
 
 
+void GenericServiceImpl::GetFlags(const GetFlagsRequestPB* req,
+  GetFlagsResponsePB* resp,
+  rpc::RpcContext* rpc) {
+  // If no tags were specified, return all flags that have non-default values.
+  // If tags were specified, also filter flags that don't match any tag.
+  bool all_flags = req->all_flags();
+  for (const auto& entry : GetFlagsMap()) {
+if (entry.second.is_default && !all_flags) {
+  continue;
+}
+unordered_set tags;
+GetFlagTags(entry.first, );
+bool matches = req->tags().empty();
+for (const auto& tag : req->tags()) {
+  if (ContainsKey(tags, tag)) {
+matches = true;
+break;
+  }
+}
+if (!matches) continue;
+
+auto* flag = resp->add_flags();
+flag->set_name(entry.first);
+flag->set_value(CheckFlagAndRedact(entry.second, EscapeMode::NONE));
+flag->set_is_default_value(entry.second.current_value ==