kudu git commit: [jepsen] updated the test script to use gradle

2018-10-17 Thread alexey 
Repository: kudu
Updated Branches:
  refs/heads/master 8386ef9e2 -> 7a4caf966


[jepsen] updated the test script to use gradle

Since maven build is gone with commit 8c184e2, it's necessary
to switch from mvn to gradlew while building and running Kudu
Jepsen tests.

In addition, java/kudu-jepsen/README.adoc is updated to reflect
changes made in fbe4312.

Change-Id: If6879c30def3f4c7d8f70945f200ff77275976dc
Reviewed-on: http://gerrit.cloudera.org:8080/11717
Reviewed-by: Grant Henke 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7a4caf96
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7a4caf96
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7a4caf96

Branch: refs/heads/master
Commit: 7a4caf9660617c8dad2f95b939adec9eb1f9fd04
Parents: 8386ef9
Author: Alexey Serbin 
Authored: Wed Oct 17 12:21:24 2018 -0700
Committer: Alexey Serbin 
Committed: Thu Oct 18 02:23:41 2018 +

--
 java/kudu-jepsen/README.adoc | 11 +--
 src/kudu/scripts/jepsen.sh   | 20 +---
 2 files changed, 10 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/7a4caf96/java/kudu-jepsen/README.adoc
--
diff --git a/java/kudu-jepsen/README.adoc b/java/kudu-jepsen/README.adoc
index 562bcdd..2339f9d 100644
--- a/java/kudu-jepsen/README.adoc
+++ b/java/kudu-jepsen/README.adoc
@@ -46,11 +46,10 @@ To build the library the following components are required:
 
 * JDK 8
 
-To build the project and install necessary Java packages into the local Maven
-repository, run in the parent directory (i.e. `$KUDU_HOME/java`)
+To build the project, run in the parent directory (i.e. `$KUDU_HOME/java`)
 [listing]
 
-$ ./gradlew -Pjepsen clean install
+$ ./gradlew clean assemble
 
 
 === Running
@@ -78,14 +77,14 @@ If using SSH authentication agent to hold the SSH key for 
DB nodes access,
 run in the current directory:
 [listing]
 
-$ ./gradlew -Pjepsen runJepsen -DtserverNodes="t0,t1,t2,t3,t4" 
-DmasterNodes="m0"
+$ ./gradlew runJepsen -DtserverNodes="t0,t1,t2,t3,t4" -DmasterNodes="m0"
 
 
 If not using SSH authentication agent, specify the SSH key location via the
 `sshKeyPath` property:
 [listing]
 
-$ ./gradlew -Pjepsen runJepsen -DtserverNodes="t0,t1,t2,t3,t4" 
-DmasterNodes="m0" -DsshKeyPath="./vm_root_id_rsa"
+$ ./gradlew runJepsen -DtserverNodes="t0,t1,t2,t3,t4" -DmasterNodes="m0" 
-DsshKeyPath="./vm_root_id_rsa"
 
 
 Note that commas (not spaces) are used to separate the names of the nodes. The
@@ -134,7 +133,7 @@ summary message below reports on 10 errors in 10 tests ran:
 21:41:42 Ran 10  tests containing 10 assertions.
 21:41:42 0 failures, 10 errors.
 
-To get more details, take a closer look at the output of `./gradlew -Pjepsen 
runJepsen`
+To get more details, take a closer look at the output of `./gradlew runJepsen`
 or at particular `jepsen.log` files in
 `$KUDU_HOME/java/kudu-jepsen/store/rw-register/` directory. A
 quick way to locate the corresponding section in the error log is to search for

http://git-wip-us.apache.org/repos/asf/kudu/blob/7a4caf96/src/kudu/scripts/jepsen.sh
--
diff --git a/src/kudu/scripts/jepsen.sh b/src/kudu/scripts/jepsen.sh
index ad3c0eb..870e8dd 100755
--- a/src/kudu/scripts/jepsen.sh
+++ b/src/kudu/scripts/jepsen.sh
@@ -25,15 +25,7 @@
 #   JAVA8_HOME  Default: /usr/lib/jvm/java-8-openjdk-amd64
 # Path to the JDK8 installation root.  Expecting to find 'java'
 # in the 'bin' sub-directory.  Java 8 or higher is required for the
-# maven-clojure-plugin.
-#
-#   M2_HOME Default: /usr/share/maven
-# Path to the maven (3.3.6 and newer is required).
-#
-#   MVN_FLAGS   Default: ""
-# Extra flags which are passed to 'mvn' when building and running Java
-# tests. This can be useful, for example, to choose a different maven
-# repository location.
+# nebula-clojure-plugin.
 #
 #   KUDU_MASTER_NODES   Default: ""
 # Set of master nodes for the Kudu cluster to run the Jepsen consistency
@@ -72,8 +64,6 @@ NUM_PROCS=$(getconf _NPROCESSORS_ONLN)
 
 BUILD_TYPE=${BUILD_TYPE:-debug}
 JAVA8_HOME=${JAVA8_HOME:-/usr/lib/jvm/java-8-openjdk-amd64}
-M2_HOME=${M2_HOME:-/usr/share/maven}
-MVN_FLAGS=${MVN_FLAGS:-}
 KUDU_MASTER_NODES=${KUDU_MASTER_NODES:-}
 KUDU_TSERVER_NODES=${KUDU_TSERVER_NODES:-}
 SSH_KEY=${SSH_KEY:-}
@@ -116,20 +106,20 @@ $SRC_ROOT/build-support/enable_devtoolset.sh 
$THIRDPARTY_BIN/cmake \
 make -j$NUM_PROCS 2>&1 | tee build.log
 
 export JAVA_HOME=$JAVA8_HOME
-export PATH=$JAVA_HOME/bin:$M2_HOME/bin:$PATH
+export PATH=$JAVA_HOME/bin:$PATH
 set -x
 
 pushd $SRC_ROOT/java
 echo
-echo "Building Kudu Java

[2/3] kudu git commit: KUDU-2245 Graceful leadership transfer

2018-10-17 Thread mpercy 
KUDU-2245 Graceful leadership transfer

This patch implements graceful leadership transfer, as described in the
original Raft thesis. It has the following steps:

1. An admin client sends a request to the tablet leader for it to
   transfer leadership. The client can indicate a specific voter that it
   wants to become the leader, or it can allow the current leader to
   choose its successor.
2. The leader receives the request and begins a leader transfer period.
   During a leader transfer period, the leader does not accept writes or
   config change requests. This allows followers to catch up to the
   leader. A background timer expires the transfer period after one
   election timeout, since clients should be able to ride over
   interruptions in service lasting at least that long. If another
   request to transfer leadership is received during a transfer period,
   it will be rejected.
3. During the transfer period, the leader continues to update peers.
   When it receives a response from a peer, it checks if that peer is
   a voter and fully caught up to the leader's log. If it is, and if it
   is the designated successor if one was provided, the leader signals
   the peer to start an election, which it should win. If no eligible
   successor appears, the transfer period expires and the leader resumes
   normal operation.

This is an improvement over the current leader step down method, which
causes the leader to simply relinquish leadership and snooze its
election timer for an extra long period, so another voter will likely
become leader. Leadership transfer should usually be much faster and it
allows the client to select the new leader among current voters.
However, note that it does not provide strictly better guarantees- it is
still possible that leadership will not be transferred.

I ran TestRepeatLeaderStepDown and TestGracefulLeaderStepDown 1000 times
and 200 times each, in debug and TSAN modes, with 4 stress threads, and
saw no failures.

Change-Id: Ic97343af9eb349556424c999799ed5e2941f0083
Reviewed-on: http://gerrit.cloudera.org:8080/11251
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e61de497
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e61de497
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e61de497

Branch: refs/heads/master
Commit: e61de497c487f33fe1a99c9df6dfb697e8efb757
Parents: f0ed88b
Author: Will Berkeley 
Authored: Thu Aug 9 13:44:00 2018 -0700
Committer: Will Berkeley 
Committed: Thu Oct 18 00:44:53 2018 +

--
 src/kudu/consensus/consensus-test-util.h|  28 +-
 src/kudu/consensus/consensus.proto  |  18 +-
 src/kudu/consensus/consensus_peers.cc   |  37 +-
 src/kudu/consensus/consensus_peers.h|  31 +-
 src/kudu/consensus/consensus_queue.cc   |  60 +++
 src/kudu/consensus/consensus_queue.h|  25 +-
 src/kudu/consensus/peer_manager.cc  |  12 +
 src/kudu/consensus/peer_manager.h   |   3 +
 src/kudu/consensus/raft_consensus.cc| 100 +
 src/kudu/consensus/raft_consensus.h |  33 +-
 .../integration-tests/raft_consensus-itest.cc   |  68 
 src/kudu/tools/kudu-admin-test.cc   | 365 +--
 src/kudu/tools/kudu-tool-test.cc|   2 +-
 src/kudu/tools/tool_action_tablet.cc|  57 ++-
 src/kudu/tools/tool_replica_util.cc |  14 +
 src/kudu/tools/tool_replica_util.h  |  22 +-
 src/kudu/tserver/tablet_service.cc  |  33 +-
 17 files changed, 840 insertions(+), 68 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/e61de497/src/kudu/consensus/consensus-test-util.h
--
diff --git a/src/kudu/consensus/consensus-test-util.h 
b/src/kudu/consensus/consensus-test-util.h
index d6dc3f0..a0a0e42 100644
--- a/src/kudu/consensus/consensus-test-util.h
+++ b/src/kudu/consensus/consensus-test-util.h
@@ -227,6 +227,12 @@ class DelayablePeerProxy : public TestPeerProxy {
this, kUpdate));
   }
 
+  virtual Status StartElection(const RunLeaderElectionRequestPB* /*request*/,
+   RunLeaderElectionResponsePB* /*response*/,
+   rpc::RpcController* /*controller*/) override {
+return Status::OK();
+  }
+
   virtual void RequestConsensusVoteAsync(const VoteRequestPB* request,
  VoteResponsePB* response,
  rpc::RpcController* controller,
@@ -291,6 +297,12 @@ class MockedPeerProxy : public TestPeerProxy {
 return RegisterCallbackAndRespond(kRequestVote, callback);
   }
 
+  Status

[1/3] kudu git commit: Follow-up improvements to locate_row

2018-10-17 Thread mpercy 
Repository: kudu
Updated Branches:
  refs/heads/master 117e7d906 -> 8386ef9e2


Follow-up improvements to locate_row

- Added test coverage for a string literal with a space in it.
- Abbreviated the tests by adding a couple of lambdas.
- Print out all tablet ids when multiple tablets match the primary key.
  This should never happen, but if it does this output will be useful.
- Couple of other small, non-functional changes.

Change-Id: I9f4cca87963eee1d6da45fcbd3c27f782c35e6f6
Reviewed-on: http://gerrit.cloudera.org:8080/11715
Reviewed-by: Adar Dembo 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/f0ed88bb
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/f0ed88bb
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/f0ed88bb

Branch: refs/heads/master
Commit: f0ed88bbe728d7a739a0a0ecce35aaf7535f4933
Parents: 117e7d9
Author: Will Berkeley 
Authored: Wed Oct 17 14:27:48 2018 -0700
Committer: Will Berkeley 
Committed: Thu Oct 18 00:18:25 2018 +

--
 src/kudu/tools/kudu-admin-test.cc   | 194 +--
 src/kudu/tools/tool_action_table.cc |  20 +++-
 2 files changed, 71 insertions(+), 143 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/f0ed88bb/src/kudu/tools/kudu-admin-test.cc
--
diff --git a/src/kudu/tools/kudu-admin-test.cc 
b/src/kudu/tools/kudu-admin-test.cc
index d68fb4b..ecf6323 100644
--- a/src/kudu/tools/kudu-admin-test.cc
+++ b/src/kudu/tools/kudu-admin-test.cc
@@ -1502,7 +1502,7 @@ TEST_F(AdminCliTest, TestLocateRow) {
   }, , );
   ASSERT_TRUE(s.ok()) << ToolRunInfo(s, stdout, stderr);
 
-  // Grab list of tablet_ids from any tserver and check the output.
+  // Grab list of tablet_ids from the tserver and check the output.
   vector tservers;
   vector tablet_ids;
   AppendValuesFromMap(tablet_servers_, );
@@ -1512,45 +1512,30 @@ TEST_F(AdminCliTest, TestLocateRow) {
   ASSERT_EQ(1, tablet_ids.size());
   ASSERT_STR_CONTAINS(stdout, tablet_ids[0]);
 
-  // Test a couple of error cases.
+  // Test a few error cases.
+  const auto check_bad_input = [&](const string& json, const string& error) {
+string out, err;
+Status s = RunKuduTool({
+  "table",
+  "locate_row",
+  cluster_->master()->bound_rpc_addr().ToString(),
+  kTableId,
+  json,
+}, , );
+ASSERT_TRUE(s.IsRuntimeError());
+ASSERT_STR_CONTAINS(err, error);
+  };
+
   // String instead of int.
-  stdout.clear();
-  stderr.clear();
-  s = RunKuduTool({
-"table",
-"locate_row",
-cluster_->master()->bound_rpc_addr().ToString(),
-kTableId,
-"[\"foo\"]"
-  }, , );
-  ASSERT_TRUE(s.IsRuntimeError());
-  ASSERT_STR_CONTAINS(stderr, "unable to parse");
+  NO_FATALS(check_bad_input("[\"foo\"]", "unable to parse"));
 
   // Float instead of int.
-  stdout.clear();
-  stderr.clear();
-  s = RunKuduTool({
-"table",
-"locate_row",
-cluster_->master()->bound_rpc_addr().ToString(),
-kTableId,
-"[1.2]"
-  }, , );
-  ASSERT_TRUE(s.IsRuntimeError());
-  ASSERT_STR_CONTAINS(stderr, "unable to parse");
+  NO_FATALS(check_bad_input("[1.2]", "unable to parse"));
 
   // Overflow (recall the key is INT32).
-  stdout.clear();
-  stderr.clear();
-  s = RunKuduTool({
-"table",
-"locate_row",
-cluster_->master()->bound_rpc_addr().ToString(),
-kTableId,
-Substitute("[$0]", std::to_string(std::numeric_limits::max()))
-  }, , );
-  ASSERT_TRUE(s.IsRuntimeError());
-  ASSERT_STR_CONTAINS(stderr, "out of range");
+  NO_FATALS(check_bad_input(
+  Substitute("[$0]", std::to_string(std::numeric_limits::max())),
+  "out of range"));
 }
 
 TEST_F(AdminCliTest, TestLocateRowMore) {
@@ -1608,7 +1593,7 @@ TEST_F(AdminCliTest, TestLocateRowMore) {
 "locate_row",
 cluster_->master()->bound_rpc_addr().ToString(),
 kAnotherTableId,
-"[\"foo\",0]"
+"[\"foo bar\",0]"
   }, , );
   ASSERT_TRUE(s.ok()) << ToolRunInfo(s, stdout, stderr);
   StripWhiteSpace();
@@ -1633,117 +1618,50 @@ TEST_F(AdminCliTest, TestLocateRowMore) {
   << "expected to find tablet id " << tablet_id_for_2;
   ASSERT_NE(tablet_id_for_0, tablet_id_for_2);
 
+  // Test a few error cases.
+  const auto check_bad_input = [&](const string& json, const string& error) {
+string out, err;
+Status s = RunKuduTool({
+  "table",
+  "locate_row",
+  cluster_->master()->bound_rpc_addr().ToString(),
+  kAnotherTableId,
+  json,
+}, , );
+ASSERT_TRUE(s.IsRuntimeError());
+ASSERT_STR_CONTAINS(err, error);
+  };
+
   // Test locating a row lying in a non-covered range.
-  stdout.clear();
-  stderr.clear();
-  s = RunKuduTool({
-"table",
-"locate_row",
-cluster_->master()->bound_rpc_addr().ToString(),
-

[3/3] kudu git commit: client: add timeout duration and scan attempts to scanner errors

2018-10-17 Thread mpercy 
client: add timeout duration and scan attempts to scanner errors

Previously, scanner errors could be confusing because certain types of
errors, like ServiceUnavailable, are retriable, and the RPC timeouts
mentioned in some error messages tend to be confusingly short because
after several retries the client-side deadline is looming. This patch
adds additional details to scanner timeout error messages, including
information about number of retries and the client-set timeout value.

An example of a new "enhanced" error message looks something like this
(from the new test that injects an error on scan):

Timed out: exceeded configured scan timeout of 1.000s: after 7 scan attempts: 
unable to retry before timeout: Remote error: Service unavailable: Injecting 
service unavailable status on Scan due to 
--scanner_inject_service_unavailable_on_continue_scan

Change-Id: I3a8f731f029132d0894355098d5804840f09e7c2
Reviewed-on: http://gerrit.cloudera.org:8080/11646
Reviewed-by: Adar Dembo 
Tested-by: Mike Percy 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/8386ef9e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/8386ef9e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/8386ef9e

Branch: refs/heads/master
Commit: 8386ef9e2962ee5692c0e646fdd63f9826b3b47c
Parents: e61de49
Author: Mike Percy 
Authored: Tue Oct 9 17:15:46 2018 -0700
Committer: Mike Percy 
Committed: Thu Oct 18 02:08:50 2018 +

--
 src/kudu/client/client-test.cc  | 20 
 src/kudu/client/scanner-internal.cc | 27 +++
 src/kudu/client/scanner-internal.h  |  4 
 src/kudu/tserver/tablet_service.cc  | 10 ++
 4 files changed, 53 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/8386ef9e/src/kudu/client/client-test.cc
--
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index dc7e7b8..cc4c9ab 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -114,6 +114,7 @@ DECLARE_bool(fail_dns_resolution);
 DECLARE_bool(log_inject_latency);
 DECLARE_bool(master_support_connect_to_master_rpc);
 DECLARE_bool(rpc_trace_negotiation);
+DECLARE_bool(scanner_inject_service_unavailable_on_continue_scan);
 DECLARE_int32(flush_threshold_mb);
 DECLARE_int32(flush_threshold_secs);
 DECLARE_int32(heartbeat_interval_ms);
@@ -5161,6 +5162,25 @@ TEST_F(ClientTest, 
TestLastErrorEmbeddedInScanTimeoutStatus) {
   ASSERT_STR_CONTAINS(s.ToString(), "Illegal state: Tablet not RUNNING");
 }
 
+TEST_F(ClientTest, TestRetriesEmbeddedInScanTimeoutStatus) {
+  // For the random() calls that take place during scan retries.
+  SeedRandom();
+
+  NO_FATALS(InsertTestRows(client_table_.get(), FLAGS_test_scan_num_rows));
+
+  // Allow creating a scanner but fail all of the read calls.
+  FLAGS_scanner_inject_service_unavailable_on_continue_scan = true;
+
+  // The scanner will return a retriable error on read, and we will eventually
+  // observe a timeout.
+  KuduScanner scanner(client_table_.get());
+  ASSERT_OK(scanner.SetTimeoutMillis(1000));
+  Status s = scanner.Open();
+  ASSERT_TRUE(s.IsTimedOut()) << s.ToString();
+  ASSERT_STR_CONTAINS(s.ToString(), "exceeded configured scan timeout of");
+  ASSERT_STR_CONTAINS(s.ToString(), "scan attempts");
+}
+
 TEST_F(ClientTest, TestNoDefaultPartitioning) {
 unique_ptr table_creator(client_->NewTableCreator());
 Status s = 
table_creator->table_name("TestNoDefaultPartitioning").schema(_).Create();

http://git-wip-us.apache.org/repos/asf/kudu/blob/8386ef9e/src/kudu/client/scanner-internal.cc
--
diff --git a/src/kudu/client/scanner-internal.cc 
b/src/kudu/client/scanner-internal.cc
index 51cdca6..9ac1a86 100644
--- a/src/kudu/client/scanner-internal.cc
+++ b/src/kudu/client/scanner-internal.cc
@@ -81,6 +81,20 @@ KuduScanner::Data::Data(KuduTable* table)
 KuduScanner::Data::~Data() {
 }
 
+Status KuduScanner::Data::EnrichStatusMessage(Status s) const {
+  if (scan_attempts_ > 1) {
+s = s.CloneAndPrepend(Substitute("after $0 scan attempts", 
scan_attempts_));
+  }
+  if (s.IsTimedOut()) {
+s = s.CloneAndPrepend(Substitute("exceeded configured scan timeout of $0",
+ configuration_.timeout().ToString()));
+  }
+  if (!last_error_.ok()) {
+s = s.CloneAndAppend(last_error_.ToString());
+  }
+  return s;
+}
+
 Status KuduScanner::Data::HandleError(const ScanRpcStatus& err,
   const MonoTime& deadline,
   set* blacklist,
@@ -92,10 +106,10 @@ Status KuduScanner::Data::HandleError(const ScanRpcStatus& 
err,
   // If we timed out because of

kudu git commit: client: add tablet id to scanner error messages

2018-10-17 Thread mpercy 
Repository: kudu
Updated Branches:
  refs/heads/master 082bbfe20 -> 117e7d906


client: add tablet id to scanner error messages

Some of the existing scanner log messages are confusing because at some
point the KuduScanner::Data::DebugString() method was changed to have
different output. This patch cleans up the output so that the sentences
make sense and so we always print the id of the tablet that returned the
error, if it's available.

Also fixed a couple of clang-tidy warnings.

Change-Id: I349d624c87d924df81d34c5eeb556433eac7b440
Reviewed-on: http://gerrit.cloudera.org:8080/11645
Tested-by: Mike Percy 
Reviewed-by: Will Berkeley 
Reviewed-by: Adar Dembo 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/117e7d90
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/117e7d90
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/117e7d90

Branch: refs/heads/master
Commit: 117e7d9065961cebeb86addba1262d3f2741bcc8
Parents: 082bbfe
Author: Mike Percy 
Authored: Tue Oct 9 17:15:12 2018 -0700
Committer: Mike Percy 
Committed: Wed Oct 17 23:42:05 2018 +

--
 src/kudu/client/client.cc   |  5 +++--
 src/kudu/client/scanner-internal.cc | 16 
 src/kudu/client/scanner-internal.h  | 16 +---
 3 files changed, 20 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/117e7d90/src/kudu/client/client.cc
--
diff --git a/src/kudu/client/client.cc b/src/kudu/client/client.cc
index 9230a30..1986257 100644
--- a/src/kudu/client/client.cc
+++ b/src/kudu/client/client.cc
@@ -147,6 +147,7 @@ struct tm;
 
 namespace kudu {
 class simple_spinlock;
+
 namespace client {
 
 class ResourceMetrics;
@@ -1559,13 +1560,13 @@ Status KuduScanner::NextBatch(KuduScanBatch* batch) {
   bool needs_reopen = false;
   Status s = data_->HandleError(result, batch_deadline, , 
_reopen);
   if (!s.ok()) {
-LOG(WARNING) << "Scan at tablet server " << data_->ts_->ToString() << 
" of tablet "
+LOG(WARNING) << "Scan on tablet server " << data_->ts_->ToString() << 
" with "
  << data_->DebugString() << " failed: " << 
result.status.ToString();
 return s;
   }
 
   if (data_->configuration().is_fault_tolerant()) {
-LOG(WARNING) << "Attempting to retry scan of tablet " << 
data_->DebugString()
+LOG(WARNING) << "Attempting to retry " << data_->DebugString()
  << " elsewhere.";
 return data_->ReopenCurrentTablet(batch_deadline, );
   }

http://git-wip-us.apache.org/repos/asf/kudu/blob/117e7d90/src/kudu/client/scanner-internal.cc
--
diff --git a/src/kudu/client/scanner-internal.cc 
b/src/kudu/client/scanner-internal.cc
index af8f783..51cdca6 100644
--- a/src/kudu/client/scanner-internal.cc
+++ b/src/kudu/client/scanner-internal.cc
@@ -30,6 +30,7 @@
 
 #include "kudu/client/client-internal.h"
 #include "kudu/client/meta_cache.h"
+#include "kudu/client/schema.h"
 #include "kudu/common/common.pb.h"
 #include "kudu/common/encoded_key.h"
 #include "kudu/common/partition.h"
@@ -203,9 +204,17 @@ void KuduScanner::Data::UpdateResourceMetrics() {
   }
 }
 
+string KuduScanner::Data::DebugString() const {
+  return Substitute("Scanner { table: $0, tablet: $1, projection: $2, 
scan_spec: $3 }",
+table_->name(),
+remote_ ? remote_->tablet_id() : "",
+configuration_.projection()->ToString(),
+configuration_.spec().ToString(*table_->schema().schema_));
+}
+
 ScanRpcStatus KuduScanner::Data::AnalyzeResponse(const Status& rpc_status,
  const MonoTime& 
overall_deadline,
- const MonoTime& deadline) {
+ const MonoTime& rpc_deadline) 
{
   if (rpc_status.ok() && !last_response_.has_error()) {
 return ScanRpcStatus{ScanRpcStatus::OK, Status::OK()};
   }
@@ -239,11 +248,10 @@ ScanRpcStatus KuduScanner::Data::AnalyzeResponse(const 
Status& rpc_status,
 }
 
 if (rpc_status.IsTimedOut()) {
-  if (overall_deadline == deadline) {
+  if (overall_deadline == rpc_deadline) {
 return ScanRpcStatus{ScanRpcStatus::OVERALL_DEADLINE_EXCEEDED, 
rpc_status};
-  } else {
-return ScanRpcStatus{ScanRpcStatus::RPC_DEADLINE_EXCEEDED, rpc_status};
   }
+  return ScanRpcStatus{ScanRpcStatus::RPC_DEADLINE_EXCEEDED, rpc_status};
 }
 return ScanRpcStatus{ScanRpcStatus::RPC_ERROR, rpc_status};
   }

http://git-wip-us.apache.org/repos/asf/kudu/blob/117e7d90/src/kudu/client/scanner-internal.h

[2/3] kudu git commit: thrift operators: add more test coverage

2018-10-17 Thread danburkert 
thrift operators: add more test coverage

This is a follow-up to Adar's suggestions in
https://gerrit.cloudera.org/c/11693/.

Change-Id: Ifdba996a9c4e6e01f55c257bfd10fb3f74f0f894
Reviewed-on: http://gerrit.cloudera.org:8080/11713
Reviewed-by: Adar Dembo 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/1e8ef0ea
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/1e8ef0ea
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/1e8ef0ea

Branch: refs/heads/master
Commit: 1e8ef0eac6d411a225bd3a4101a7e19346eb1048
Parents: 7607e23
Author: Dan Burkert 
Authored: Wed Oct 17 12:53:05 2018 -0700
Committer: Dan Burkert 
Committed: Wed Oct 17 23:02:28 2018 +

--
 src/kudu/sentry/thrift_operators-test.cc | 56 +--
 1 file changed, 36 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/1e8ef0ea/src/kudu/sentry/thrift_operators-test.cc
--
diff --git a/src/kudu/sentry/thrift_operators-test.cc 
b/src/kudu/sentry/thrift_operators-test.cc
index 4ef3873..d4aa070 100644
--- a/src/kudu/sentry/thrift_operators-test.cc
+++ b/src/kudu/sentry/thrift_operators-test.cc
@@ -65,32 +65,48 @@ TEST(ThriftOperatorsTest, TestOperatorLt) {
   ASSERT_EQ(2, groups.size()) << groups;
 
   // TSentryPrivilege::operator<
+  const string kServer = "server1";
+  const string kDatabase = "db1";
+
   TSentryPrivilege db_priv;
-  db_priv.__set_serverName("server1");
-  db_priv.__set_dbName("db1");
+  db_priv.__set_serverName(kServer);
+  db_priv.__set_dbName(kDatabase);
 
-  TSentryPrivilege tbl_priv;
-  tbl_priv.__set_serverName("server1");
-  tbl_priv.__set_dbName("db1");
-  tbl_priv.__set_tableName("tbl1");
+  TSentryPrivilege tbl1_priv;
+  tbl1_priv.__set_serverName(kServer);
+  tbl1_priv.__set_dbName(kDatabase);
+  tbl1_priv.__set_tableName("tbl1");
 
-  NO_FATALS(AssertCompareRequirements(db_priv, tbl_priv));
-  set privileges { db_priv, tbl_priv };
-  ASSERT_EQ(2, privileges.size()) << privileges;
+  TSentryPrivilege tbl2_priv;
+  tbl2_priv.__set_serverName(kServer);
+  tbl2_priv.__set_dbName(kDatabase);
+  tbl2_priv.__set_tableName("tbl2");
 
+  NO_FATALS(AssertCompareRequirements(db_priv, tbl1_priv));
+  NO_FATALS(AssertCompareRequirements(db_priv, tbl2_priv));
+  NO_FATALS(AssertCompareRequirements(tbl1_priv, tbl2_priv));
+  set privileges { db_priv, tbl1_priv, tbl2_priv };
+  ASSERT_EQ(3, privileges.size()) << privileges;
 
   // TSentryAuthorizable::operator<
   TSentryAuthorizable db_authorizable;
-  db_authorizable.__set_server("server1");
-  db_authorizable.__set_db("db1");
-
-  TSentryAuthorizable tbl_authorizable;
-  tbl_authorizable.__set_server("server1");
-  tbl_authorizable.__set_db("db1");
-  tbl_authorizable.__set_table("tbl1");
-
-  NO_FATALS(AssertCompareRequirements(db_authorizable, tbl_authorizable));
-  set authorizables { db_authorizable, tbl_authorizable };
-  ASSERT_EQ(2, authorizables.size()) << authorizables;
+  db_authorizable.__set_server(kServer);
+  db_authorizable.__set_db(kDatabase);
+
+  TSentryAuthorizable tbl1_authorizable;
+  tbl1_authorizable.__set_server(kServer);
+  tbl1_authorizable.__set_db(kDatabase);
+  tbl1_authorizable.__set_table("tbl1");
+
+  TSentryAuthorizable tbl2_authorizable;
+  tbl2_authorizable.__set_server(kServer);
+  tbl2_authorizable.__set_db(kDatabase);
+  tbl2_authorizable.__set_table("tbl2");
+
+  NO_FATALS(AssertCompareRequirements(db_authorizable, tbl1_authorizable));
+  NO_FATALS(AssertCompareRequirements(db_authorizable, tbl2_authorizable));
+  NO_FATALS(AssertCompareRequirements(tbl1_authorizable, tbl2_authorizable));
+  set authorizables { db_authorizable, tbl1_authorizable, 
tbl2_authorizable };
+  ASSERT_EQ(3, authorizables.size()) << authorizables;
 }
 } // namespace sentry

[1/3] kudu git commit: sentry module: introduce using declarations for generated types

2018-10-17 Thread danburkert 
Repository: kudu
Updated Branches:
  refs/heads/master 36d09f187 -> 082bbfe20


sentry module: introduce using declarations for generated types

This cleans up the .cc files in the sentry module with 'using'
declarations, as suggested by Adar.

Change-Id: I51f61eedc1069d362fa1ce26ce0449cce8f0846b
Reviewed-on: http://gerrit.cloudera.org:8080/11712
Reviewed-by: Adar Dembo 
Reviewed-by: Hao Hao 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7607e235
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7607e235
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7607e235

Branch: refs/heads/master
Commit: 7607e2354bf61957e4d3168d75d4990169dc2472
Parents: 36d09f1
Author: Dan Burkert 
Authored: Wed Oct 17 12:46:03 2018 -0700
Committer: Dan Burkert 
Committed: Wed Oct 17 23:02:24 2018 +

--
 src/kudu/sentry/sentry_client-test.cc| 55 ---
 src/kudu/sentry/sentry_client.cc | 55 ---
 src/kudu/sentry/thrift_operators-test.cc | 34 +++--
 3 files changed, 79 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/7607e235/src/kudu/sentry/sentry_client-test.cc
--
diff --git a/src/kudu/sentry/sentry_client-test.cc 
b/src/kudu/sentry/sentry_client-test.cc
index d8c845d..10a4134 100644
--- a/src/kudu/sentry/sentry_client-test.cc
+++ b/src/kudu/sentry/sentry_client-test.cc
@@ -35,6 +35,17 @@
 #include "kudu/util/test_macros.h"
 #include "kudu/util/test_util.h"
 
+using sentry::TAlterSentryRoleAddGroupsRequest;
+using sentry::TAlterSentryRoleAddGroupsResponse;
+using sentry::TAlterSentryRoleGrantPrivilegeRequest;
+using sentry::TAlterSentryRoleGrantPrivilegeResponse;
+using sentry::TCreateSentryRoleRequest;
+using sentry::TDropSentryRoleRequest;
+using sentry::TListSentryPrivilegesRequest;
+using sentry::TListSentryPrivilegesResponse;
+using sentry::TSentryAuthorizable;
+using sentry::TSentryGroup;
+using sentry::TSentryPrivilege;
 using std::set;
 using std::string;
 using std::unique_ptr;
@@ -106,12 +117,12 @@ TEST_P(SentryClientTest, TestMiniSentryLifecycle) {
  sentry_client_opts));
   auto smoketest = [&]() -> Status {
 return client.Execute([](SentryClient* client) -> Status {
-::sentry::TCreateSentryRoleRequest create_req;
+TCreateSentryRoleRequest create_req;
 create_req.requestorUserName = "test-admin";
 create_req.roleName = "test-role";
 RETURN_NOT_OK(client->CreateRole(create_req));
 
-::sentry::TDropSentryRoleRequest drop_req;
+TDropSentryRoleRequest drop_req;
 drop_req.requestorUserName = "test-admin";
 drop_req.roleName = "test-role";
 RETURN_NOT_OK(client->DropRole(drop_req));
@@ -137,7 +148,7 @@ TEST_P(SentryClientTest, TestMiniSentryLifecycle) {
 TEST_P(SentryClientTest, TestCreateDropRole) {
 
   { // Create a role
-::sentry::TCreateSentryRoleRequest req;
+TCreateSentryRoleRequest req;
 req.requestorUserName = "test-admin";
 req.roleName = "viewer";
 ASSERT_OK(sentry_client_->CreateRole(req));
@@ -148,7 +159,7 @@ TEST_P(SentryClientTest, TestCreateDropRole) {
   }
 
   { // Attempt to create a role as a non-admin user.
-::sentry::TCreateSentryRoleRequest req;
+TCreateSentryRoleRequest req;
 req.requestorUserName = "joe-interloper";
 req.roleName = "fuzz";
 Status s = sentry_client_->CreateRole(req);
@@ -156,7 +167,7 @@ TEST_P(SentryClientTest, TestCreateDropRole) {
   }
 
   { // Attempt to drop the role as a non-admin user.
-::sentry::TDropSentryRoleRequest req;
+TDropSentryRoleRequest req;
 req.requestorUserName = "joe-interloper";
 req.roleName = "viewer";
 Status s = sentry_client_->DropRole(req);
@@ -164,7 +175,7 @@ TEST_P(SentryClientTest, TestCreateDropRole) {
   }
 
   { // Drop the role
-::sentry::TDropSentryRoleRequest req;
+TDropSentryRoleRequest req;
 req.requestorUserName = "test-admin";
 req.roleName = "viewer";
 ASSERT_OK(sentry_client_->DropRole(req));
@@ -180,15 +191,15 @@ TEST_P(SentryClientTest, TestCreateDropRole) {
 // instances.
 TEST_P(SentryClientTest, TestListPrivilege) {
   // Attempt to access Sentry privileges by a non admin user.
-  ::sentry::TSentryAuthorizable authorizable;
+  TSentryAuthorizable authorizable;
   authorizable.server = "server";
   authorizable.db = "db";
   authorizable.table = "table";
-  ::sentry::TListSentryPrivilegesRequest request;
+  TListSentryPrivilegesRequest request;
   request.requestorUserName = "joe-interloper";
   request.authorizableHierarchy = authorizable;
   request.__set_principalName("viewer");
-  ::sentry::TListSentryPrivilegesResponse response;
+

[3/3] kudu git commit: HMS integration: provide Java API to override owner during table create

2018-10-17 Thread danburkert 
HMS integration: provide Java API to override owner during table create

This is necessary so that Impala can override the table owner, instead
of using the inferred username of the connection (which would be
'impala'). This API is marked as unstable since it's not clear we will
want to support it long term, and there are some open questions about
the fine-grained privileges required to use it.

Change-Id: I21069d81a4b54e55c399933b69789bf83bd58de0
Reviewed-on: http://gerrit.cloudera.org:8080/11413
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/082bbfe2
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/082bbfe2
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/082bbfe2

Branch: refs/heads/master
Commit: 082bbfe2048279518a9103e4edcd91f815705294
Parents: 1e8ef0e
Author: Dan Burkert 
Authored: Mon Sep 10 16:02:54 2018 -0700
Committer: Dan Burkert 
Committed: Wed Oct 17 23:02:37 2018 +

--
 java/kudu-client/build.gradle   |  1 +
 .../apache/kudu/client/CreateTableOptions.java  | 23 +++
 .../client/TestHiveMetastoreIntegration.java| 72 
 src/kudu/master/catalog_manager.cc  |  4 +-
 src/kudu/master/master.proto|  4 ++
 5 files changed, 102 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/082bbfe2/java/kudu-client/build.gradle
--
diff --git a/java/kudu-client/build.gradle b/java/kudu-client/build.gradle
index 077e050..363135b 100644
--- a/java/kudu-client/build.gradle
+++ b/java/kudu-client/build.gradle
@@ -37,6 +37,7 @@ dependencies {
 
   testCompile project(":kudu-test-utils")
   testCompile libs.hamcrestCore
+  testCompile libs.hiveMetastore
   testCompile libs.junit
   testCompile libs.log4j
   testCompile libs.mockitoCore

http://git-wip-us.apache.org/repos/asf/kudu/blob/082bbfe2/java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java
--
diff --git 
a/java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java 
b/java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java
index cb2ba52..2f137a8 100644
--- 
a/java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java
+++ 
b/java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java
@@ -19,6 +19,8 @@ package org.apache.kudu.client;
 
 import java.util.List;
 
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import org.apache.yetus.audience.InterfaceAudience;
@@ -209,6 +211,27 @@ public class CreateTableOptions {
 return this;
   }
 
+  /**
+   * Set the table owner as the provided username in configured external 
catalogs
+   * such as the Hive Metastore. Overrides the default of the currently 
logged-in
+   * username or Kerberos principal.
+   *
+   * This is an unstable method because it is not yet clear whether this should
+   * be supported directly in the long run, rather than requiring the table 
creator
+   * to re-assign ownership explicitly.
+   *
+   * @param owner the username to set as the table owner in external catalogs
+   * @return this instance
+   */
+  @InterfaceAudience.LimitedPrivate("Impala")
+  @InterfaceStability.Unstable
+  public CreateTableOptions setOwner(String owner) {
+Preconditions.checkArgument(!Strings.isNullOrEmpty(owner),
+"table owner must not be null or empty");
+pb.setOwner(owner);
+return this;
+  }
+
   Master.CreateTableRequestPB.Builder getBuilder() {
 if (!splitRows.isEmpty() || !rangePartitions.isEmpty()) {
   pb.setSplitRowsRangeBounds(new Operation.OperationsEncoder()

http://git-wip-us.apache.org/repos/asf/kudu/blob/082bbfe2/java/kudu-client/src/test/java/org/apache/kudu/client/TestHiveMetastoreIntegration.java
--
diff --git 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestHiveMetastoreIntegration.java
 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestHiveMetastoreIntegration.java
new file mode 100644
index 000..daf2396
--- /dev/null
+++ 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestHiveMetastoreIntegration.java
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0

kudu git commit: ksck: quell build warning

2018-10-17 Thread adar 
Repository: kudu
Updated Branches:
  refs/heads/master fbe43129f -> 36d09f187


ksck: quell build warning

Was seeing:
../../src/kudu/tools/ksck_checksum.cc:677:1: warning: control reaches end of 
non-void function [-Wreturn-type]

Change-Id: I696ff790111cc0b371dbb762a3c5167d834136f7
Reviewed-on: http://gerrit.cloudera.org:8080/11661
Reviewed-by: Will Berkeley 
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/36d09f18
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/36d09f18
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/36d09f18

Branch: refs/heads/master
Commit: 36d09f1876a7a1c9bd27f71b194e61aebd5237c9
Parents: fbe4312
Author: Andrew Wong 
Authored: Thu Oct 11 14:38:56 2018 -0700
Committer: Adar Dembo 
Committed: Wed Oct 17 22:28:13 2018 +

--
 src/kudu/tools/ksck_checksum.cc | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/36d09f18/src/kudu/tools/ksck_checksum.cc
--
diff --git a/src/kudu/tools/ksck_checksum.cc b/src/kudu/tools/ksck_checksum.cc
index f41efc7..c87211c 100644
--- a/src/kudu/tools/ksck_checksum.cc
+++ b/src/kudu/tools/ksck_checksum.cc
@@ -674,6 +674,8 @@ Status KsckChecksummer::ChecksumData(const 
KsckChecksumOptions& opts,
   num_results, num_replicas);
   return s;
   }
+  LOG(FATAL) << "Reached unreachable section";
+  return Status::RuntimeError("Unreachable statement"); // Suppress compiler 
warnings.
 }
 
 } // namespace tools

[3/5] kudu git commit: KUDU-2411: (Part 1) Break out existing test utilities into a seperate module

2018-10-17 Thread granthenke 
http://git-wip-us.apache.org/repos/asf/kudu/blob/15f1416f/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java
--
diff --git 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java
index 17ed3db..66f9e71 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java
@@ -13,10 +13,10 @@
  */
 package org.apache.kudu.client;
 
-import static org.apache.kudu.util.AssertHelpers.assertEventuallyTrue;
-import static org.apache.kudu.util.ClientTestUtil.createBasicSchemaInsert;
-import static org.apache.kudu.util.ClientTestUtil.getBasicCreateTableOptions;
-import static org.apache.kudu.util.ClientTestUtil.getBasicSchema;
+import static org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue;
+import static org.apache.kudu.test.ClientTestUtil.createBasicSchemaInsert;
+import static org.apache.kudu.test.ClientTestUtil.getBasicCreateTableOptions;
+import static org.apache.kudu.test.ClientTestUtil.getBasicSchema;
 import static org.junit.Assert.assertNotNull;
 
 import java.io.Closeable;
@@ -30,12 +30,14 @@ import java.util.concurrent.TimeUnit;
 import javax.security.auth.Subject;
 
 import org.apache.kudu.client.Client.AuthenticationCredentialsPB;
-import org.apache.kudu.client.MiniKuduCluster.MiniKuduClusterBuilder;
-import org.apache.kudu.junit.RetryRule;
+import org.apache.kudu.test.cluster.MiniKuduCluster;
+import org.apache.kudu.test.cluster.MiniKuduCluster.MiniKuduClusterBuilder;
+import org.apache.kudu.test.junit.RetryRule;
 import org.apache.kudu.master.Master.ConnectToMasterResponsePB;
-import org.apache.kudu.util.AssertHelpers;
-import org.apache.kudu.util.AssertHelpers.BooleanExpression;
-import org.apache.kudu.util.CapturingLogAppender;
+import org.apache.kudu.test.cluster.FakeDNS;
+import org.apache.kudu.test.junit.AssertHelpers;
+import org.apache.kudu.test.junit.AssertHelpers.BooleanExpression;
+import org.apache.kudu.test.CapturingLogAppender;
 import org.apache.kudu.util.SecurityUtil;
 import org.hamcrest.CoreMatchers;
 import org.junit.After;

http://git-wip-us.apache.org/repos/asf/kudu/blob/15f1416f/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurityContextRealUser.java
--
diff --git 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurityContextRealUser.java
 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurityContextRealUser.java
index a1aaa8e..d4dc753 100644
--- 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurityContextRealUser.java
+++ 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurityContextRealUser.java
@@ -17,7 +17,7 @@
 
 package org.apache.kudu.client;
 
-import org.apache.kudu.client.MiniKuduCluster.MiniKuduClusterBuilder;
+import org.apache.kudu.test.cluster.MiniKuduCluster.MiniKuduClusterBuilder;
 import org.apache.kudu.test.KuduTestHarness;
 import org.junit.Before;
 import org.junit.Rule;
@@ -25,8 +25,8 @@ import org.junit.Test;
 
 import java.util.ArrayList;
 
-import static org.apache.kudu.util.ClientTestUtil.getBasicSchema;
-import static org.apache.kudu.util.ClientTestUtil.scanTableToStrings;
+import static org.apache.kudu.test.ClientTestUtil.getBasicSchema;
+import static org.apache.kudu.test.ClientTestUtil.scanTableToStrings;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;

http://git-wip-us.apache.org/repos/asf/kudu/blob/15f1416f/java/kudu-client/src/test/java/org/apache/kudu/client/TestServerInfo.java
--
diff --git 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestServerInfo.java 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestServerInfo.java
index 2d82272..233ae17 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestServerInfo.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestServerInfo.java
@@ -15,6 +15,7 @@ package org.apache.kudu.client;
 
 import java.net.InetAddress;
 
+import org.apache.kudu.test.cluster.FakeDNS;
 import org.junit.Assert;
 import org.junit.Test;
 

http://git-wip-us.apache.org/repos/asf/kudu/blob/15f1416f/java/kudu-client/src/test/java/org/apache/kudu/client/TestStatistics.java
--
diff --git 
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestStatistics.java 
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestStatistics.java
index f9cf0ca..1dfa319 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestStatistics.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestStatistics.java
@@ -16,9 +16,9 @@
 //

[5/5] kudu git commit: [java] Remove the kudu-jepsen -Pjepsen build flag

2018-10-17 Thread granthenke 
[java] Remove the kudu-jepsen -Pjepsen build flag

This removes the “jepsen” property required to build
the kudu-jepsen module. This was added back when
the build was Java 7 compatible and kudu-jepsen
required Java 8. Now the build requires Java 8 and
removing this will ensure we don’t break compiling
kudu-jepsen.

Change-Id: I82ec4470c18fb44ae71e8f6439b10fb63dde9d50
Reviewed-on: http://gerrit.cloudera.org:8080/11709
Reviewed-by: Alexey Serbin 
Reviewed-by: Adar Dembo 
Tested-by: Grant Henke 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/fbe43129
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/fbe43129
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/fbe43129

Branch: refs/heads/master
Commit: fbe43129fe8edbb8a504996e9155b62aa65d336b
Parents: 15f1416
Author: Grant Henke 
Authored: Wed Oct 17 14:24:04 2018 -0500
Committer: Grant Henke 
Committed: Wed Oct 17 20:41:04 2018 +

--
 java/kudu-jepsen/build.gradle | 9 -
 1 file changed, 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/fbe43129/java/kudu-jepsen/build.gradle
--
diff --git a/java/kudu-jepsen/build.gradle b/java/kudu-jepsen/build.gradle
index 4fe1bf4..8b99944 100644
--- a/java/kudu-jepsen/build.gradle
+++ b/java/kudu-jepsen/build.gradle
@@ -17,15 +17,6 @@
 
 apply plugin: "nebula.clojure"
 
-// Skip kudu-jepsen module tasks unless "Jepsen" property is passed.
-if (!project.hasProperty("jepsen")) {
-  gradle.taskGraph.whenReady {
-gradle.taskGraph.allTasks.each {
-  it.onlyIf { it.project != project }
-}
-  }
-}
-
 repositories {
   maven { url "http://clojars.org/repo/; }
 }

[2/5] kudu git commit: KUDU-2411: (Part 1) Break out existing test utilities into a seperate module

2018-10-17 Thread granthenke 
http://git-wip-us.apache.org/repos/asf/kudu/blob/15f1416f/java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java
--
diff --git 
a/java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java 
b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java
new file mode 100644
index 000..9b67a9b
--- /dev/null
+++ 
b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java
@@ -0,0 +1,376 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.kudu.test;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.stumbleupon.async.Callback;
+import com.stumbleupon.async.Deferred;
+import org.apache.kudu.ColumnSchema;
+import org.apache.kudu.ColumnTypeAttributes;
+import org.apache.kudu.Schema;
+import org.apache.kudu.Type;
+import org.apache.kudu.client.AsyncKuduClient;
+import org.apache.kudu.client.AsyncKuduScanner;
+import org.apache.kudu.client.AsyncKuduSession;
+import org.apache.kudu.client.CreateTableOptions;
+import org.apache.kudu.client.Insert;
+import org.apache.kudu.client.KuduClient;
+import org.apache.kudu.client.KuduException;
+import org.apache.kudu.client.KuduPredicate;
+import org.apache.kudu.client.KuduScanToken;
+import org.apache.kudu.client.KuduScanner;
+import org.apache.kudu.client.KuduSession;
+import org.apache.kudu.client.KuduTable;
+import org.apache.kudu.client.PartialRow;
+import org.apache.kudu.client.RowResult;
+import org.apache.kudu.client.RowResultIterator;
+import org.apache.kudu.util.DecimalUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Utilities useful for cluster testing.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public abstract class ClientTestUtil {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(ClientTestUtil.class);
+
+  public static final Callback defaultErrorCB = new 
Callback() {
+@Override
+public Object call(Object arg) throws Exception {
+  if (arg == null) {
+return null;
+  }
+  if (arg instanceof Exception) {
+LOG.warn("Got exception", (Exception) arg);
+  } else {
+LOG.warn("Got an error response back {}", arg);
+  }
+  return new Exception("cannot recover from error: " + arg);
+}
+  };
+
+  /**
+   * Counts the rows from the {@code scanner} until exhaustion. It doesn't 
require the scanner to
+   * be new, so it can be used to finish scanning a previously-started scan.
+   */
+  public static int countRowsInScan(AsyncKuduScanner scanner, long timeoutMs) 
throws Exception {
+final AtomicInteger counter = new AtomicInteger();
+
+Callback cb = new Callback() {
+  @Override
+  public Object call(RowResultIterator arg) throws Exception {
+if (arg == null) return null;
+counter.addAndGet(arg.getNumRows());
+return null;
+  }
+};
+
+while (scanner.hasMoreRows()) {
+  Deferred data = scanner.nextRows();
+  data.addCallbacks(cb, defaultErrorCB);
+  data.join(timeoutMs);
+}
+return counter.get();
+  }
+
+  /**
+   * Same as {@link #countRowsInScan(AsyncKuduScanner, long)}, but defaults 
the timeout to 60
+   * seconds.
+   */
+  public static int countRowsInScan(AsyncKuduScanner scanner) throws Exception 
{
+return countRowsInScan(scanner, 6);
+  }
+
+  public static int countRowsInScan(KuduScanner scanner) throws KuduException {
+int counter = 0;
+while (scanner.hasMoreRows()) {
+  counter += scanner.nextRows().getNumRows();
+}
+return counter;
+  }
+
+  /**
+   * Scans the table and returns the number of rows.
+   * @param table the table
+   * @param predicates optional predicates to apply to the scan
+   * @return the number of rows in the table matching the predicates
+

[4/5] kudu git commit: KUDU-2411: (Part 1) Break out existing test utilities into a seperate module

2018-10-17 Thread granthenke 
KUDU-2411: (Part 1) Break out existing test utilities into a seperate module

This patch breaks out the test harness and test
util classes into a seperate module.

This cleans up the dependencies on test jars and
is a preliminary step to a public test utility. Though
all of the classes still remain marked as private
until the interfaces are fleshed out further.

Change-Id: Ifa935d6038b6d8756b332178347cec5cb70660a9
Reviewed-on: http://gerrit.cloudera.org:8080/11686
Reviewed-by: Dan Burkert 
Reviewed-by: Adar Dembo 
Tested-by: Grant Henke 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/15f1416f
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/15f1416f
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/15f1416f

Branch: refs/heads/master
Commit: 15f1416f67dcb714842d02647a1f2e06e675660d
Parents: 7072a85
Author: Grant Henke 
Authored: Mon Oct 15 13:34:12 2018 -0500
Committer: Grant Henke 
Committed: Wed Oct 17 20:40:42 2018 +

--
 java/kudu-backup/build.gradle   |   2 +-
 .../org/apache/kudu/backup/TestKuduBackup.scala |   2 +-
 java/kudu-client-tools/build.gradle |   2 +-
 .../kudu/mapreduce/tools/ITExportCsv.java   |   2 +-
 .../kudu/mapreduce/tools/ITImportCsv.java   |   4 +-
 .../kudu/mapreduce/tools/ITImportParquet.java   |   4 +-
 .../tools/ITImportParquetPreCheck.java  |   2 +-
 .../kudu/mapreduce/tools/ITRowCounter.java  |   2 +-
 java/kudu-client/build.gradle   |  16 +-
 .../java/org/apache/kudu/client/FakeDNS.java| 192 --
 .../java/org/apache/kudu/client/ITClient.java   |   6 +-
 .../org/apache/kudu/client/ITClientStress.java  |   8 +-
 .../kudu/client/ITScannerMultiTablet.java   |   2 +-
 .../org/apache/kudu/client/MiniKuduCluster.java | 642 --
 .../org/apache/kudu/client/TestAlterTable.java  |   4 +-
 .../apache/kudu/client/TestAsyncKuduClient.java |  10 +-
 .../kudu/client/TestAsyncKuduSession.java   |  10 +-
 .../kudu/client/TestAuthnTokenReacquire.java|  10 +-
 .../client/TestAuthnTokenReacquireOpen.java |   8 +-
 .../kudu/client/TestClientFailoverSupport.java  |  14 +-
 .../kudu/client/TestConnectToCluster.java   |   1 +
 .../apache/kudu/client/TestConnectionCache.java |   3 +-
 .../apache/kudu/client/TestHandleTooBusy.java   |   6 +-
 .../org/apache/kudu/client/TestHybridTime.java  |   4 +-
 .../org/apache/kudu/client/TestKuduClient.java  |  22 +-
 .../org/apache/kudu/client/TestKuduSession.java |  12 +-
 .../org/apache/kudu/client/TestKuduTable.java   |  12 +-
 .../apache/kudu/client/TestLeaderFailover.java  |   8 +-
 .../apache/kudu/client/TestMasterFailover.java  |   6 +-
 .../apache/kudu/client/TestMiniKuduCluster.java | 130 
 .../kudu/client/TestMultipleLeaderFailover.java |  12 +-
 .../org/apache/kudu/client/TestNegotiation.java |   6 +-
 .../org/apache/kudu/client/TestPartialRow.java  |   2 +-
 .../apache/kudu/client/TestRemoteTablet.java|   2 +-
 .../org/apache/kudu/client/TestRowErrors.java   |   8 +-
 .../org/apache/kudu/client/TestRowResult.java   |   4 +-
 .../org/apache/kudu/client/TestScanToken.java   |   8 +-
 .../kudu/client/TestScannerMultiTablet.java |   2 +-
 .../org/apache/kudu/client/TestSecurity.java|  20 +-
 .../client/TestSecurityContextRealUser.java |   6 +-
 .../org/apache/kudu/client/TestServerInfo.java  |   1 +
 .../org/apache/kudu/client/TestStatistics.java  |   6 +-
 .../org/apache/kudu/client/TestTimeouts.java|   6 +-
 .../java/org/apache/kudu/junit/RetryRule.java   |  83 ---
 .../org/apache/kudu/test/KuduTestHarness.java   | 445 -
 .../org/apache/kudu/util/AssertHelpers.java |  46 --
 .../apache/kudu/util/CapturingLogAppender.java  |  82 ---
 .../org/apache/kudu/util/ClientTestUtil.java| 375 ---
 .../org/apache/kudu/util/KuduBinaryLocator.java |  94 ---
 .../org/apache/kudu/util/ProtobufUtils.java |  64 --
 .../java/org/apache/kudu/util/RandomUtils.java  |  49 --
 java/kudu-flume-sink/build.gradle   |   2 +-
 .../sink/AvroKuduOperationsProducerTest.java|   2 +-
 .../sink/KeyedKuduOperationsProducerTest.java   |   2 +-
 .../apache/kudu/flume/sink/KuduSinkTest.java|   2 +-
 ...expKuduOperationsProducerParseErrorTest.java |   3 +-
 .../sink/RegexpKuduOperationsProducerTest.java  |   2 +-
 .../kudu/flume/sink/SecureKuduSinkTest.java |   4 +-
 java/kudu-hive/build.gradle |   2 +-
 .../hive/metastore/TestKuduMetastorePlugin.java |   2 +-
 java/kudu-jepsen/build.gradle   |   2 +-
 java/kudu-mapreduce/build.gradle|   2 +-
 .../apache/kudu/mapreduce/ITInputFormatJob.java |   4 +-
 .../kudu/mapreduce/ITKuduTableInputFormat.java  |   4 +-
 .../kudu/mapreduce/ITKuduTableOutputFormat.java |   6 +-
 .../kudu/mapreduce/ITOutputFormatJob.java   |   6 +-

[1/5] kudu git commit: Fix flakiness in MultiThreadedRpcTest.TestBlowOutServiceQueue

2018-10-17 Thread granthenke 
Repository: kudu
Updated Branches:
  refs/heads/master 58ecdd155 -> fbe43129f


Fix flakiness in MultiThreadedRpcTest.TestBlowOutServiceQueue

Commit eded05723e changed the error message returned by Socket::Recv
when EOF is received from the remote. The corresponding string used for
matching in TestBlowOutServiceQueue was not updated, so in rare
circumstances the test would fail because it saw an unexpected error
message.

Change-Id: I952f4a01ecf73de5c92c6b0820b9176dde0a5a8f
Reviewed-on: http://gerrit.cloudera.org:8080/11705
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo 


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7072a85c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7072a85c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7072a85c

Branch: refs/heads/master
Commit: 7072a85c5e0cb872c3369b80903ec5a978e92e1b
Parents: 58ecdd1
Author: Will Berkeley 
Authored: Tue Oct 16 17:59:41 2018 -0700
Committer: Will Berkeley 
Committed: Wed Oct 17 04:10:54 2018 +

--
 src/kudu/rpc/mt-rpc-test.cc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/kudu/blob/7072a85c/src/kudu/rpc/mt-rpc-test.cc
--
diff --git a/src/kudu/rpc/mt-rpc-test.cc b/src/kudu/rpc/mt-rpc-test.cc
index 7427850..d349e1d 100644
--- a/src/kudu/rpc/mt-rpc-test.cc
+++ b/src/kudu/rpc/mt-rpc-test.cc
@@ -195,7 +195,7 @@ void IncrementBackpressureOrShutdown(const Status* status, 
int* backpressure, in
 ++(*backpressure);
   } else if (msg.find("shutting down") != string::npos) {
 ++(*shutdown);
-  } else if (msg.find("got EOF from remote") != string::npos) {
+  } else if (msg.find("recv got EOF from") != string::npos) {
 ++(*shutdown);
   } else {
 FAIL() << "Unexpected status message: " << msg;