kudu git commit: Fix thrift operator< implementations
Repository: kudu
Updated Branches:
refs/heads/master 09848055b -> 58ecdd155
Fix thrift operator< implementations
The previous implementation didn't account for unset optional fields,
and did not short-circuit after finding the first field which was
smaller. Buggy operator< is problematic because the std::map and
std::set types require the operator< to maintain a total ordering; if it
doesn't then the set and map invariants will not hold.
Change-Id: I1aacf602c603b05433c357a4a236ba0b9e521392
Reviewed-on: http://gerrit.cloudera.org:8080/11693
Reviewed-by: Hao Hao
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/58ecdd15
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/58ecdd15
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/58ecdd15
Branch: refs/heads/master
Commit: 58ecdd15553178b25612f65de71ef25d07058079
Parents: 0984805
Author: Dan Burkert
Authored: Mon Oct 15 17:44:27 2018 -0700
Committer: Dan Burkert
Committed: Wed Oct 17 00:01:56 2018 +
--
src/kudu/sentry/CMakeLists.txt | 1 +
src/kudu/sentry/thrift_operators-test.cc | 102 ++
src/kudu/sentry/thrift_operators.cc | 38 ++
3 files changed, 127 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/58ecdd15/src/kudu/sentry/CMakeLists.txt
--
diff --git a/src/kudu/sentry/CMakeLists.txt b/src/kudu/sentry/CMakeLists.txt
index 3222793..fee3855 100644
--- a/src/kudu/sentry/CMakeLists.txt
+++ b/src/kudu/sentry/CMakeLists.txt
@@ -78,4 +78,5 @@ if (NOT NO_TESTS)
ADD_KUDU_TEST(sentry_action-test)
ADD_KUDU_TEST(sentry_client-test)
+ ADD_KUDU_TEST(thrift_operators-test)
endif()
http://git-wip-us.apache.org/repos/asf/kudu/blob/58ecdd15/src/kudu/sentry/thrift_operators-test.cc
--
diff --git a/src/kudu/sentry/thrift_operators-test.cc
b/src/kudu/sentry/thrift_operators-test.cc
new file mode 100644
index 000..46bd6c8
--- /dev/null
+++ b/src/kudu/sentry/thrift_operators-test.cc
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include
+#include
+
+#include
+#include
+
+#include "kudu/sentry/sentry_policy_service_types.h"
+#include "kudu/util/test_macros.h"
+#include "kudu/util/test_util.h"
+
+using std::string;
+using std::set;
+
+namespace kudu {
+namespace sentry {
+
+class ThriftOperatorsTest : public KuduTest {
+};
+
+template
+void AssertCompareRequirements(const T& a, const T& b) {
+ // Values must not be less than themselves.
+ ASSERT_FALSE(a < a) << a;
+ ASSERT_FALSE(b < b) << b;
+
+ // Two values may not be simultaneously less than each other.
+ if (a < b) {
+ASSERT_FALSE(b < a);
+ }
+}
+
+TEST_F(ThriftOperatorsTest, TestOperatorLt) {
+ // TSentryRole::operator<
+ ::sentry::TSentryRole role_a;
+ role_a.__set_roleName("a");
+
+ ::sentry::TSentryRole role_b;
+ role_b.__set_roleName("b");
+
+ NO_FATALS(AssertCompareRequirements(role_a, role_b));
+ set<::sentry::TSentryRole> roles { role_a, role_b };
+ ASSERT_EQ(2, roles.size()) << roles;
+
+ // TSentryGroup::operator<
+ ::sentry::TSentryGroup group_a;
+ group_a.__set_groupName("a");
+
+ ::sentry::TSentryGroup group_b;
+ group_b.__set_groupName("b");
+
+ NO_FATALS(AssertCompareRequirements(group_a, group_b));
+ set<::sentry::TSentryGroup> groups { group_a, group_b };
+ ASSERT_EQ(2, groups.size()) << groups;
+
+ // TSentryPrivilege::operator<
+ ::sentry::TSentryPrivilege db_priv;
+ db_priv.__set_serverName("server1");
+ db_priv.__set_dbName("db1");
+
+ ::sentry::TSentryPrivilege tbl_priv;
+ tbl_priv.__set_serverName("server1");
+ tbl_priv.__set_dbName("db1");
+ tbl_priv.__set_tableName("tbl1");
+
+ NO_FATALS(AssertCompareRequirements(db_priv, tbl_priv));
+ set<::sentry::TSentryPrivilege> privileges { db_priv, tbl_priv };
+ ASSERT_EQ(2, privileges.size()) << privileges;
+
+
+ // TSentryAuthorizable::operator<
+
[1/2] kudu git commit: [compaction] KUDU-2056: Expose a metric for how much a tablet needs to be compacted
Repository: kudu
Updated Branches:
refs/heads/master 8c184e2a9 -> 09848055b
[compaction] KUDU-2056: Expose a metric for how much a tablet needs to be
compacted
This adds a new metric 'average_diskrowset_height' that reflects how
uncompacted a tablet replica is. This metric is obtained by integrating
the height function with respect to the by-data-size probability
distribution used by the compaction policy.
To implement the integration, I piggy-backed on the function that
computes the CDF for the rowset layout, since computing the integral
requires computing the CDF and it seemed wasteful to first compute the
CDF, then go through an almost entirely similar bit of logic to compute
the average height.
Change-Id: I98493b901d37bb278167ba2fe98d322a86a1f0f9
Reviewed-on: http://gerrit.cloudera.org:8080/11639
Reviewed-by: Andrew Wong
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d3684a7b
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d3684a7b
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d3684a7b
Branch: refs/heads/master
Commit: d3684a7b2add8f06b7189adb9ce9222b8ae1eff5
Parents: 8c184e2
Author: Will Berkeley
Authored: Mon Oct 8 16:09:41 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 16 20:47:13 2018 +
--
src/kudu/tablet/compaction_policy-test.cc | 166 +
src/kudu/tablet/compaction_policy.cc | 2 +-
src/kudu/tablet/rowset_info.cc| 101 +--
src/kudu/tablet/rowset_info.h | 19 +--
src/kudu/tablet/tablet.cc | 33 -
src/kudu/tablet/tablet_metrics.cc | 10 +-
src/kudu/tablet/tablet_metrics.h | 9 +-
7 files changed, 287 insertions(+), 53 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/d3684a7b/src/kudu/tablet/compaction_policy-test.cc
--
diff --git a/src/kudu/tablet/compaction_policy-test.cc
b/src/kudu/tablet/compaction_policy-test.cc
index 4277f9d..8ec0829 100644
--- a/src/kudu/tablet/compaction_policy-test.cc
+++ b/src/kudu/tablet/compaction_policy-test.cc
@@ -21,6 +21,7 @@
#include
#include
#include
+#include
#include
#include
@@ -33,6 +34,7 @@
#include "kudu/tablet/compaction_policy.h"
#include "kudu/tablet/mock-rowsets.h"
#include "kudu/tablet/rowset.h"
+#include "kudu/tablet/rowset_info.h"
#include "kudu/tablet/rowset_tree.h"
#include "kudu/util/env.h"
#include "kudu/util/faststring.h"
@@ -276,5 +278,169 @@ TEST_F(TestCompactionPolicy, KUDU2251) {
ASSERT_LT(quality, 2.0);
}
+namespace {
+double ComputeAverageRowsetHeight(
+const vector>& intervals) {
+ RowSetVector rowsets;
+ for (const auto& interval : intervals) {
+rowsets.push_back(std::make_shared(interval.first,
+ interval.second));
+ }
+ RowSetTree tree;
+ CHECK_OK(tree.Reset(rowsets));
+
+ double avg_height;
+ RowSetInfo::ComputeCdfAndCollectOrdered(tree, _height, nullptr, nullptr);
+ return avg_height;
+}
+} // anonymous namespace
+
+class KeySpaceCdfTest : public KuduTest {
+ protected:
+ static void AssertWithinEpsilon(double epsilon,
+ double expected,
+ double actual) {
+ASSERT_GE(actual, expected - epsilon);
+ASSERT_LE(actual, expected + epsilon);
+ }
+};
+
+// Test the computation of average rowset heights. This isn't strictly used in
+// compaction policy, but this is a convenient place to put it, for now.
+TEST_F(KeySpaceCdfTest, TestComputingAverageRowSetHeight) {
+ // No rowsets.
+ EXPECT_EQ(0.0, ComputeAverageRowsetHeight({ }));
+
+ /* A rowset that's one key wide.
+ * |
+ */
+ EXPECT_EQ(0.0, ComputeAverageRowsetHeight({ { "A", "A" } }));
+
+ /* A single rowset.
+ * [ --- ]
+ */
+ EXPECT_EQ(1.0, ComputeAverageRowsetHeight({ { "A", "B" } }));
+
+ /* Two rowsets with no empty space between.
+ * [ --- ][ --- ]
+ */
+ EXPECT_EQ(1.0, ComputeAverageRowsetHeight({ { "A", "B" }, { "B", "C" } }));
+
+
+ /* Three rowsets with no empty spaces between.
+ * [ --- ][ --- ][ --- ]
+ */
+ EXPECT_EQ(1.0, ComputeAverageRowsetHeight({ { "A", "B" },
+ { "B", "C" },
+ { "C", "D" } }));
+
+ /* Two rowsets with empty space between them.
+ * [ --- ] [ --- ]
+ */
+ EXPECT_EQ(1.0, ComputeAverageRowsetHeight({ { "A", "B" }, { "C", "D" } }));
+
+ /* Three rowsets with empty space between them.
+ * [ --- ] [ --- ] [ --- ]
+ */
+ EXPECT_EQ(1.0, ComputeAverageRowsetHeight({ { "A", "B" },
+ { "C", "D" },
+ { "E",
[2/2] kudu git commit: Update Java README Intellij test runner instructions
Update Java README Intellij test runner instructions Change-Id: Ie372417350f50a7d4f5567cd991ac5d886329cdb Reviewed-on: http://gerrit.cloudera.org:8080/11700 Reviewed-by: Grant Henke Tested-by: Grant Henke Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/09848055 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/09848055 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/09848055 Branch: refs/heads/master Commit: 09848055b089be98754b82f6cae74ddc78f6 Parents: d3684a7 Author: Dan Burkert Authored: Tue Oct 16 13:31:40 2018 -0700 Committer: Dan Burkert Committed: Tue Oct 16 20:58:27 2018 + -- java/README.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/09848055/java/README.adoc -- diff --git a/java/README.adoc b/java/README.adoc index 5049b51..caccea8 100644 --- a/java/README.adoc +++ b/java/README.adoc @@ -140,7 +140,8 @@ to import the Gradle project. For the most consistent behavior on the command line and in the IDE, be sure to enable `Delegate IDE build/run actions to gradle` -as described https://www.jetbrains.com/help/idea/gradle.html#delegate_build_gradle[here]. +and run tests using the `Gradle Test Runner` as described +https://www.jetbrains.com/help/idea/gradle.html#delegate_build_gradle[here]. === Eclipse
[2/5] kudu git commit: [tools] Add locate row tool
[tools] Add locate row tool
Sometimes while debugging I find it frustrating that it's very difficult
to tell what tablet a particular row belongs to. This basic tool
provides a way to find out, by providing a simple interface that accepts
a primary key as a JSON array and will print out the tablet id of the
corresponding tablet, or an error if there is no such tablet.
For example, with a table created like
CREATE TABLE test (
key0 STRING NOT NULL,
key1 INT32 NOT NULL,
PRIMARY KEY(key0, key1)
)
an invocation of the tool looks like
$ kudu table locate_row localhost:7053 test "[\"foo\", 2]"
The choice of a JSON array is a compromise between CSV, which is easiest
and fastest to type in the common case, and a verbose JSON object like
{ "key0" : "foo", "key1" : 2 }, which is most explicit but long and
difficult to type on the command line. A JSON array has the benefit of
having well-defined escaping rules and formatting while being almost as
easy to type out as CSV.
A note about tests: it's difficult to verify the answer of the tool
independently. However, the implementation is just scan tokens, so the
tablet-finding logic should be well-exercised by lots of other client
tests. Consequently, the tests focus on error cases and sanity checks.
Change-Id: Idcdcf10bfe6b9df686e86b7134e8634fc0efaac3
Reviewed-on: http://gerrit.cloudera.org:8080/11666
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/fa5a0db5
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/fa5a0db5
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/fa5a0db5
Branch: refs/heads/master
Commit: fa5a0db5a21c0f9d6554857ce6641b95b670f25e
Parents: e3b1e05
Author: Will Berkeley
Authored: Fri Oct 12 01:20:12 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 16 19:39:06 2018 +
--
src/kudu/tools/kudu-admin-test.cc | 264 +++
src/kudu/tools/tool_action_table.cc | 128 ++-
2 files changed, 391 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/fa5a0db5/src/kudu/tools/kudu-admin-test.cc
--
diff --git a/src/kudu/tools/kudu-admin-test.cc
b/src/kudu/tools/kudu-admin-test.cc
index 046e8db..edb0c4d 100644
--- a/src/kudu/tools/kudu-admin-test.cc
+++ b/src/kudu/tools/kudu-admin-test.cc
@@ -21,6 +21,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -48,6 +49,7 @@
#include "kudu/gutil/gscoped_ptr.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/strings/split.h"
+#include "kudu/gutil/strings/strip.h"
#include "kudu/gutil/strings/substitute.h"
#include "kudu/integration-tests/cluster_itest_util.h"
#include "kudu/integration-tests/cluster_verifier.h"
@@ -1478,5 +1480,267 @@ TEST_F(AdminCliTest, TestDescribeTable) {
")\n"
"REPLICAS 1");
}
+
+TEST_F(AdminCliTest, TestLocateRow) {
+ FLAGS_num_tablet_servers = 1;
+ FLAGS_num_replicas = 1;
+
+ NO_FATALS(BuildAndStart());
+
+ // Test an OK case. Not much going on here since the table has only one
+ // tablet, which covers the whole universe.
+ string stdout, stderr;
+ Status s = RunKuduTool({
+"table",
+"locate_row",
+cluster_->master()->bound_rpc_addr().ToString(),
+kTableId,
+"[-1]"
+ }, , );
+ ASSERT_TRUE(s.ok()) << ToolRunInfo(s, stdout, stderr);
+
+ // Grab list of tablet_ids from any tserver and check the output.
+ vector tservers;
+ vector tablet_ids;
+ AppendValuesFromMap(tablet_servers_, );
+ ListRunningTabletIds(tservers.front(),
+ MonoDelta::FromSeconds(30),
+ _ids);
+ ASSERT_EQ(1, tablet_ids.size());
+ ASSERT_STR_CONTAINS(stdout, tablet_ids[0]);
+
+ // Test a couple of error cases.
+ // String instead of int.
+ stdout.clear();
+ stderr.clear();
+ s = RunKuduTool({
+"table",
+"locate_row",
+cluster_->master()->bound_rpc_addr().ToString(),
+kTableId,
+"[\"foo\"]"
+ }, , );
+ ASSERT_TRUE(s.IsRuntimeError());
+ ASSERT_STR_CONTAINS(stderr, "unable to parse");
+
+ // Float instead of int.
+ stdout.clear();
+ stderr.clear();
+ s = RunKuduTool({
+"table",
+"locate_row",
+cluster_->master()->bound_rpc_addr().ToString(),
+kTableId,
+"[1.2]"
+ }, , );
+ ASSERT_TRUE(s.IsRuntimeError());
+ ASSERT_STR_CONTAINS(stderr, "unable to parse");
+
+ // Overflow (recall the key is INT32).
+ stdout.clear();
+ stderr.clear();
+ s = RunKuduTool({
+"table",
+"locate_row",
+cluster_->master()->bound_rpc_addr().ToString(),
+kTableId,
+Substitute("[$0]", std::to_string(std::numeric_limits::max()))
+ }, , );
+ ASSERT_TRUE(s.IsRuntimeError());
+ ASSERT_STR_CONTAINS(stderr, "out of range");
+}
+
[5/5] kudu git commit: [Java] Remove the Maven Build
[Java] Remove the Maven Build
This patch removes the Maven poms and remaining
references to the Maven build. Going forward the
Gradle build will be used.
Change-Id: If3c1b6b5bc2218a6285bde10d9ab7e1d9013a023
Reviewed-on: http://gerrit.cloudera.org:8080/11667
Reviewed-by: Dan Burkert
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/8c184e2a
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/8c184e2a
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/8c184e2a
Branch: refs/heads/master
Commit: 8c184e2a972729cbb026b6962d6bf558b86e9cf5
Parents: acf1dee
Author: Grant Henke
Authored: Fri Oct 12 10:54:57 2018 -0500
Committer: Grant Henke
Committed: Tue Oct 16 19:55:53 2018 +
--
RELEASING.adoc| 7 -
build-support/jenkins/build-and-test.sh | 119 +++-
build-support/jenkins/post-build-clean.sh | 2 +-
docs/developing.adoc | 4 +-
docs/installation.adoc| 3 +-
docs/support/scripts/make_site.sh | 8 +-
java/README.adoc | 29 --
java/kudu-backup/pom.xml | 338 ---
java/kudu-client-tools/pom.xml| 159 ---
java/kudu-client/pom.xml | 305
java/kudu-flume-sink/pom.xml | 139 --
java/kudu-hive/pom.xml| 116
java/kudu-jepsen/README.adoc | 22 +-
java/kudu-jepsen/pom.xml | 150 --
java/kudu-mapreduce/pom.xml | 133 -
java/kudu-spark-tools/pom.xml | 203 --
java/kudu-spark/pom.xml | 177
java/pom.xml | 368 -
18 files changed, 60 insertions(+), deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/8c184e2a/RELEASING.adoc
--
diff --git a/RELEASING.adoc b/RELEASING.adoc
index 4f68b75..40cbbcd 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -85,13 +85,6 @@ branch with the same name and the previously-noted SHA1.
. Check out the `master` branch and bump the version in `version.txt`.
-. Update the version for the Java client from within the `java` directory:
-+
-
- cd java
- mvn versions:set -DnewVersion=1.x.y-SNAPSHOT
-
-
. Update the version in `java/gradle.properties`.
. If the python API has changed since the previous release, bump the Python
version
http://git-wip-us.apache.org/repos/asf/kudu/blob/8c184e2a/build-support/jenkins/build-and-test.sh
--
diff --git a/build-support/jenkins/build-and-test.sh
b/build-support/jenkins/build-and-test.sh
index 6ba91c9..b74c2ef 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -57,12 +57,6 @@
# BUILD_JAVADefault: 1
# Build and test java code if this is set to 1.
#
-# BUILD_MAVEN Default: 0
-# When building java code, build with Maven if this is set to 1.
-#
-# BUILD_GRADLE Default: 1
-# When building java code, build with Gradle if this is set to 1.
-#
# BUILD_PYTHON Default: 1
# Build and test the Python wrapper of the client API.
#
@@ -75,11 +69,6 @@
# Extra flags which are passed to 'pip install' when setting up the build
# environment for the Python wrapper.
#
-# MVN_FLAGS Default: ""
-# Extra flags which are passed to 'mvn' when building and running Java
-# tests. This can be useful, for example, to choose a different maven
-# repository location.
-#
# GRADLE_FLAGS Default: ""
# Extra flags which are passed to 'gradle' when running Gradle commands.
#
@@ -120,7 +109,6 @@ export
KUDU_ALLOW_SLOW_TESTS=${KUDU_ALLOW_SLOW_TESTS:-$DEFAULT_ALLOW_SLOW_TESTS}
export KUDU_COMPRESS_TEST_OUTPUT=${KUDU_COMPRESS_TEST_OUTPUT:-1}
export TEST_TMPDIR=${TEST_TMPDIR:-/tmp/kudutest-$UID}
BUILD_JAVA=${BUILD_JAVA:-1}
-BUILD_MAVEN=${BUILD_MAVEN:-0}
BUILD_GRADLE=${BUILD_GRADLE:-1}
BUILD_PYTHON=${BUILD_PYTHON:-1}
BUILD_PYTHON3=${BUILD_PYTHON3:-1}
@@ -144,7 +132,6 @@ rm -rf $BUILD_ROOT
mkdir -p $BUILD_ROOT
# Same for the Java tests, which aren't inside BUILD_ROOT
-rm -rf $SOURCE_ROOT/java/*/target
rm -rf $SOURCE_ROOT/java/*/build
list_flaky_tests() {
@@ -167,11 +154,6 @@ if [ -n "$BUILD_ID" ]; then
trap cleanup EXIT
fi
-export TOOLCHAIN_DIR=/opt/toolchain
-if [ -d "$TOOLCHAIN_DIR" ]; then
- PATH=$TOOLCHAIN_DIR/apache-maven-3.0/bin:$PATH
-fi
-
THIRDPARTY_TYPE=
if [ "$BUILD_TYPE" = "TSAN" ]; then
THIRDPARTY_TYPE=tsan
@@ -393,47 +375,32 @@ if [ "$BUILD_JAVA" == "1" ]; then
pushd
[1/5] kudu git commit: [sentry] Fill out more sentry client API
Repository: kudu
Updated Branches:
refs/heads/master 02e82ca14 -> 8c184e2a9
[sentry] Fill out more sentry client API
This commit adds more sentry client APIs, e.g list/grant sentry
privileges as well as grant roles to groups, for the upcoming sentry
authorization provider.
Change-Id: I34695cd4cc6723b70617164d58f8681cefd09ddd
Reviewed-on: http://gerrit.cloudera.org:8080/11657
Reviewed-by: Dan Burkert
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e3b1e059
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e3b1e059
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e3b1e059
Branch: refs/heads/master
Commit: e3b1e05990c88ae2d330738f512af59202cbe87d
Parents: 02e82ca
Author: Hao Hao
Authored: Wed Oct 10 16:10:59 2018 -0700
Committer: Hao Hao
Committed: Tue Oct 16 18:06:58 2018 +
--
src/kudu/hms/hms_catalog-test.cc | 2 +-
src/kudu/sentry/sentry_client-test.cc | 221 +++--
src/kudu/sentry/sentry_client.cc | 30
src/kudu/sentry/sentry_client.h | 20 ++-
4 files changed, 229 insertions(+), 44 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/e3b1e059/src/kudu/hms/hms_catalog-test.cc
--
diff --git a/src/kudu/hms/hms_catalog-test.cc b/src/kudu/hms/hms_catalog-test.cc
index 53bd462..20f975f 100644
--- a/src/kudu/hms/hms_catalog-test.cc
+++ b/src/kudu/hms/hms_catalog-test.cc
@@ -167,8 +167,8 @@ class HmsCatalogTest : public KuduTest {
}
void TearDown() override {
-ASSERT_OK(hms_->Stop());
ASSERT_OK(hms_client_->Stop());
+ASSERT_OK(hms_->Stop());
}
Status StopHms() {
http://git-wip-us.apache.org/repos/asf/kudu/blob/e3b1e059/src/kudu/sentry/sentry_client-test.cc
--
diff --git a/src/kudu/sentry/sentry_client-test.cc
b/src/kudu/sentry/sentry_client-test.cc
index 519e277..d8c845d 100644
--- a/src/kudu/sentry/sentry_client-test.cc
+++ b/src/kudu/sentry/sentry_client-test.cc
@@ -18,6 +18,8 @@
#include "kudu/sentry/sentry_client.h"
#include
+#include
+#include
#include
#include
@@ -33,7 +35,9 @@
#include "kudu/util/test_macros.h"
#include "kudu/util/test_util.h"
+using std::set;
using std::string;
+using std::unique_ptr;
using std::vector;
namespace kudu {
@@ -45,20 +49,63 @@ class SentryClientTest : public KuduTest,
bool KerberosEnabled() const {
return GetParam();
}
+
+ void SetUp() override {
+bool enable_kerberos = KerberosEnabled();
+
+thrift::ClientOptions sentry_client_opts;
+
+sentry_.reset(new MiniSentry());
+if (enable_kerberos) {
+ kdc_.reset(new MiniKdc());
+ ASSERT_OK(kdc_->Start());
+
+ // Create a service principal for Sentry, and configure it to use it.
+ string spn = "sentry/127.0@krbtest.com";
+ string ktpath;
+ ASSERT_OK(kdc_->CreateServiceKeytab("sentry/127.0.0.1", ));
+
+ ASSERT_OK(rpc::SaslInit());
+ sentry_->EnableKerberos(kdc_->GetEnvVars()["KRB5_CONFIG"], spn, ktpath);
+
+ ASSERT_OK(kdc_->CreateUserPrincipal("kudu"));
+ ASSERT_OK(kdc_->Kinit("kudu"));
+ ASSERT_OK(kdc_->SetKrb5Environment());
+ sentry_client_opts.enable_kerberos = true;
+ sentry_client_opts.service_principal = "sentry";
+}
+ASSERT_OK(sentry_->Start());
+
+sentry_client_.reset(new SentryClient(sentry_->address(),
+ sentry_client_opts));
+ASSERT_OK(sentry_client_->Start());
+ }
+
+ void TearDown() override {
+ASSERT_OK(sentry_client_->Stop());
+ASSERT_OK(sentry_->Stop());
+ }
+
+ protected:
+ unique_ptr sentry_client_;
+ unique_ptr sentry_;
+ unique_ptr kdc_;
};
INSTANTIATE_TEST_CASE_P(KerberosEnabled, SentryClientTest, ::testing::Bool());
-TEST_F(SentryClientTest, TestMiniSentryLifecycle) {
- MiniSentry mini_sentry;
- ASSERT_OK(mini_sentry.Start());
-
+TEST_P(SentryClientTest, TestMiniSentryLifecycle) {
// Create an HA Sentry client and ensure it automatically reconnects after
service interruption.
thrift::HaClient client;
+ thrift::ClientOptions sentry_client_opts;
+ if (KerberosEnabled()) {
+sentry_client_opts.enable_kerberos = true;
+sentry_client_opts.service_principal = "sentry";
+ }
- ASSERT_OK(client.Start(vector({ mini_sentry.address() }),
thrift::ClientOptions()));
-
- auto smoketest = [&] () -> Status {
-return client.Execute([] (SentryClient* client) -> Status {
+ ASSERT_OK(client.Start(vector({sentry_->address()}),
+ sentry_client_opts));
+ auto smoketest = [&]() -> Status {
+return client.Execute([](SentryClient* client) -> Status {
[4/5] kudu git commit: [Java] Remove the Maven Build
http://git-wip-us.apache.org/repos/asf/kudu/blob/8c184e2a/java/kudu-spark/pom.xml
--
diff --git a/java/kudu-spark/pom.xml b/java/kudu-spark/pom.xml
deleted file mode 100644
index 56196f5..000
--- a/java/kudu-spark/pom.xml
+++ /dev/null
@@ -1,177 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
-4.0.0
-
-org.apache.kudu
-kudu-parent
-1.9.0-SNAPSHOT
-
-
-
kudu-${spark.version.label}_${scala.binary.version}
-Kudu Spark Bindings
-
-
-
-org.apache.kudu
-kudu-client
-${project.version}
-
-
-
-org.apache.yetus
-audience-annotations
-${yetus.version}
-
-
-
-org.apache.spark
-spark-core_${scala.binary.version}
-${spark.version}
-provided
-
-
-
-org.scala-lang
-scala-library
-
-
-
-org.scala-lang
-scalap
-
-
-
-
-org.apache.spark
-spark-sql_${scala.binary.version}
-${spark.version}
-provided
-
-
-org.scala-lang
-scala-library
-${scala.version}
-provided
-
-
-org.scala-lang
-scalap
-${scala.version}
-provided
-
-
-org.slf4j
-slf4j-api
-${slf4j.version}
-provided
-
-
-
-org.apache.kudu
-kudu-client
-${project.version}
-test-jar
-test
-
-
-org.apache.spark
-spark-sql_${scala.binary.version}
-${spark.version}
-test-jar
-test
-
-
-junit
-junit
-${junit.version}
-test
-
-
-org.scalatest
-scalatest_${scala.binary.version}
-${scalatest.version}
-test
-
-
-
-
-
-src/main/scala
-src/test/scala
-
-
-org.apache.maven.plugins
-maven-compiler-plugin
-
-
-
-net.alchim31.maven
-scala-maven-plugin
-${scala-maven-plugin.version}
-
-${project.build.sourceEncoding}
-${scala.version}
-
-
--feature
-
--Xlint
-
-
-
-
-scala-compile-first
-process-resources
-
-add-source
-compile
-
-
-
-scala-test-compile
-process-test-resources
-
-testCompile
-
-
-
-
-
-
- org.apache.maven.plugins
- maven-surefire-plugin
-
-
-
-org.apache.maven.plugins
-maven-shade-plugin
-${maven-shade-plugin.version}
-
-
-
-org.apache.kudu:kudu-client
-com.stumbleupon:async
-
-
-
-
-
-org.antipathy
-mvn-scalafmt
-
-
-
-
http://git-wip-us.apache.org/repos/asf/kudu/blob/8c184e2a/java/pom.xml
--
diff --git a/java/pom.xml b/java/pom.xml
deleted file mode 100644
index ce13307..000
--- a/java/pom.xml
+++ /dev/null
@@ -1,368 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-4.0.0
-
-org.apache.kudu
-kudu-parent
-1.9.0-SNAPSHOT
-pom
-
-
-
-org.apache
-apache
-18
-
-
-Kudu
-
-
-Kudu.
-
-
-
-
-
-UTF-8
-
UTF-8
-
-
-1.8.1
-1.8
-
[3/5] kudu git commit: [tools] Add checking for row existence to the locate_row tool
[tools] Add checking for row existence to the locate_row tool
This enhances the `kudu tablet locate_row` tool to that, in addition to
locating which tablet a row with a given primary key would end up in, it
also will check if the row actually exists when the -check_row_existence
flag is supplied. If the row does not exist, the tool returns an error;
if it does, it prints the row.
Example invocation:
$ bin/kudu table locate_row localhost:7053
default.loadgen_auto_49c97e85f6aa43b5a89723c7eee396b0 "[0]" -check_row_existence
850632cee25b43368cb6f226b53e76eb
(int64 key=0, int32 int_val=1, string
string_val="2.xx")
Change-Id: I760d3fbb2e30a3ffba7143d3b51da6f3fd62b034
Reviewed-on: http://gerrit.cloudera.org:8080/11684
Tested-by: Will Berkeley
Reviewed-by: Alexey Serbin
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/acf1dee8
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/acf1dee8
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/acf1dee8
Branch: refs/heads/master
Commit: acf1dee87ce347398ca86cc4959e38bf8794efe3
Parents: fa5a0db
Author: Will Berkeley
Authored: Mon Oct 15 01:37:30 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 16 19:41:54 2018 +
--
src/kudu/tools/kudu-admin-test.cc | 70
src/kudu/tools/tool_action_table.cc | 29 +
2 files changed, 99 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/acf1dee8/src/kudu/tools/kudu-admin-test.cc
--
diff --git a/src/kudu/tools/kudu-admin-test.cc
b/src/kudu/tools/kudu-admin-test.cc
index edb0c4d..d68fb4b 100644
--- a/src/kudu/tools/kudu-admin-test.cc
+++ b/src/kudu/tools/kudu-admin-test.cc
@@ -39,6 +39,7 @@
#include "kudu/client/client.h"
#include "kudu/client/schema.h"
#include "kudu/client/shared_ptr.h"
+#include "kudu/client/write_op.h"
#include "kudu/common/common.pb.h"
#include "kudu/common/partial_row.h"
#include "kudu/common/wire_protocol.pb.h"
@@ -79,8 +80,10 @@ DECLARE_int32(num_tablet_servers);
using kudu::client::KuduClient;
using kudu::client::KuduClientBuilder;
using kudu::client::KuduColumnSchema;
+using kudu::client::KuduInsert;
using kudu::client::KuduSchema;
using kudu::client::KuduSchemaBuilder;
+using kudu::client::KuduTable;
using kudu::client::KuduTableAlterer;
using kudu::client::KuduTableCreator;
using kudu::client::sp::shared_ptr;
@@ -1742,5 +1745,72 @@ TEST_F(AdminCliTest, TestLocateRowMore) {
ASSERT_STR_CONTAINS(stderr,
"Wrong type during field extraction: expected object
array");
}
+
+TEST_F(AdminCliTest, TestLocateRowAndCheckRowPresence) {
+ FLAGS_num_tablet_servers = 1;
+ FLAGS_num_replicas = 1;
+
+ NO_FATALS(BuildAndStart());
+
+ // Grab list of tablet_ids from any tserver so we can check the output.
+ vector tservers;
+ vector tablet_ids;
+ AppendValuesFromMap(tablet_servers_, );
+ ListRunningTabletIds(tservers.front(),
+ MonoDelta::FromSeconds(30),
+ _ids);
+ ASSERT_EQ(1, tablet_ids.size());
+ const string& expected_tablet_id = tablet_ids[0];
+
+ // Test the case when the row does not exist.
+ string stdout, stderr;
+ Status s = RunKuduTool({
+"table",
+"locate_row",
+cluster_->master()->bound_rpc_addr().ToString(),
+kTableId,
+"[0]",
+"-check_row_existence",
+ }, , );
+ ASSERT_TRUE(s.IsRuntimeError()) << ToolRunInfo(s, stdout, stderr);
+ ASSERT_STR_CONTAINS(stdout, expected_tablet_id);
+ ASSERT_STR_CONTAINS(stderr, "row does not exist");
+
+ // Insert row with key = 0.
+ client::sp::shared_ptr client;
+ CreateClient();
+ client::sp::shared_ptr table;
+ ASSERT_OK(client->OpenTable(kTableId, ));
+ unique_ptr insert(table->NewInsert());
+ auto* row = insert->mutable_row();
+ ASSERT_OK(row->SetInt32("key", 0));
+ ASSERT_OK(row->SetInt32("int_val", 12345));
+ ASSERT_OK(row->SetString("string_val", "hello"));
+ const string row_str = row->ToString();
+ auto session = client->NewSession();
+ ASSERT_OK(session->Apply(insert.release()));
+ ASSERT_OK(session->Flush());
+ ASSERT_OK(session->Close());
+
+ // Test the case when the row exists. Since the scan is done by a subprocess
+ // using a different client instance, it's possible the scan will not
+ // immediately retrieve the row even though the write has already succeeded,
+ // so we ASSERT_EVENTUALLY.
+ ASSERT_EVENTUALLY([&]() {
+stdout.clear();
+stderr.clear();
+s = RunKuduTool({
+ "table",
+ "locate_row",
+ cluster_->master()->bound_rpc_addr().ToString(),
+ kTableId,
+ "[0]",
+ "-check_row_existence",
+}, , );
+ASSERT_TRUE(s.ok()) << ToolRunInfo(s, stdout, stderr);
+ASSERT_STR_CONTAINS(stdout,
kudu git commit: Fix jsonreader signed int extraction and add unsigned extraction
Repository: kudu
Updated Branches:
refs/heads/master 614b446e1 -> 02e82ca14
Fix jsonreader signed int extraction and add unsigned extraction
JsonReader::GetInt64's implementation was actually of
JsonReader::GetUint64, and similarly for JsonReader::GetInt. This patch
corrects the error and adds JsonReader::{GetUint32, GetUint64} too.
A new test ensures that the 4 JsonReader::Get[Ui|I]nt[32|64] methods
work as expected on signed and unsigned values.
Change-Id: I0b07757bacc756643aea5613b3491a34cb051a43
Reviewed-on: http://gerrit.cloudera.org:8080/11683
Reviewed-by: Andrew Wong
Tested-by: Will Berkeley
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/02e82ca1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/02e82ca1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/02e82ca1
Branch: refs/heads/master
Commit: 02e82ca14f5c3c89f201224ea5b18dc0052ab835
Parents: 614b446
Author: Will Berkeley
Authored: Mon Oct 15 00:27:51 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 16 17:39:10 2018 +
--
src/kudu/util/jsonreader-test.cc | 104 ++
src/kudu/util/jsonreader.cc | 27 +
src/kudu/util/jsonreader.h | 8 +++
3 files changed, 139 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/02e82ca1/src/kudu/util/jsonreader-test.cc
--
diff --git a/src/kudu/util/jsonreader-test.cc b/src/kudu/util/jsonreader-test.cc
index 9f62c31..50e1fc6 100644
--- a/src/kudu/util/jsonreader-test.cc
+++ b/src/kudu/util/jsonreader-test.cc
@@ -18,9 +18,11 @@
#include "kudu/util/jsonreader.h"
#include
+#include
#include
#include
+#include // IWYU pragma: keep
#include
#include
@@ -126,6 +128,108 @@ TEST(JsonReaderTest, LessBasic) {
ASSERT_TRUE(r.ExtractObjectArray(r.root(), "bool",
nullptr).IsInvalidArgument());
}
+TEST(JsonReaderTest, SignedAndUnsignedInts) {
+ // The rapidjson code has some improper handling of the min int32 and min
+ // int64 that exposes UB.
+ #if defined(ADDRESS_SANITIZER)
+LOG(WARNING) << "this test is skipped in ASAN builds";
+return;
+ #endif
+
+ constexpr auto kMaxInt32 = std::numeric_limits::max();
+ constexpr auto kMaxInt64 = std::numeric_limits::max();
+ constexpr auto kMaxUint32 = std::numeric_limits::max();
+ constexpr auto kMaxUint64 = std::numeric_limits::max();
+ constexpr auto kMinInt32 = std::numeric_limits::min();
+ constexpr auto kMinInt64 = std::numeric_limits::min();
+ const string doc = Substitute(
+ "{ \"negative\" : -1, \"signed_big32\" : $0, \"signed_big64\" : $1, "
+ "\"unsigned_big32\" : $2, \"unsigned_big64\" : $3, "
+ "\"signed_small32\" : $4, \"signed_small64\" : $5 }",
+ kMaxInt32, kMaxInt64, kMaxUint32, kMaxUint64, kMinInt32, kMinInt64);
+ JsonReader r(doc);
+ ASSERT_OK(r.Init());
+
+ // -1.
+ const char* const negative = "negative";
+ int32_t negative32;
+ ASSERT_OK(r.ExtractInt32(r.root(), negative, ));
+ ASSERT_EQ(-1, negative32);
+ int64_t negative64;
+ ASSERT_OK(r.ExtractInt64(r.root(), negative, ));
+ ASSERT_EQ(-1, negative64);
+ ASSERT_TRUE(r.ExtractUint32(r.root(), negative,
nullptr).IsInvalidArgument());
+ ASSERT_TRUE(r.ExtractUint64(r.root(), negative,
nullptr).IsInvalidArgument());
+
+ // Max signed 32-bit integer.
+ const char* const signed_big32 = "signed_big32";
+ int32_t signed_big32_int32;
+ ASSERT_OK(r.ExtractInt32(r.root(), signed_big32, _big32_int32));
+ ASSERT_EQ(kMaxInt32, signed_big32_int32);
+ int64_t signed_big32_int64;
+ ASSERT_OK(r.ExtractInt64(r.root(), signed_big32, _big32_int64));
+ ASSERT_EQ(kMaxInt32, signed_big32_int64);
+ uint32_t signed_big32_uint32;
+ ASSERT_OK(r.ExtractUint32(r.root(), signed_big32, _big32_uint32));
+ ASSERT_EQ(kMaxInt32, signed_big32_uint32);
+ uint64_t signed_big32_uint64;
+ ASSERT_OK(r.ExtractUint64(r.root(), signed_big32, _big32_uint64));
+ ASSERT_EQ(kMaxInt32, signed_big32_uint64);
+
+ // Max signed 64-bit integer.
+ const char* const signed_big64 = "signed_big64";
+ ASSERT_TRUE(r.ExtractInt32(r.root(), signed_big64,
nullptr).IsInvalidArgument());
+ int64_t signed_big64_int64;
+ ASSERT_OK(r.ExtractInt64(r.root(), signed_big64, _big64_int64));
+ ASSERT_EQ(kMaxInt64, signed_big64_int64);
+ ASSERT_TRUE(r.ExtractUint32(r.root(), signed_big64,
nullptr).IsInvalidArgument());
+ uint64_t signed_big64_uint64;
+ ASSERT_OK(r.ExtractUint64(r.root(), signed_big64, _big64_uint64));
+ ASSERT_EQ(kMaxInt64, signed_big64_uint64);
+
+ // Max unsigned 32-bit integer.
+ const char* const unsigned_big32 = "unsigned_big32";
+ ASSERT_TRUE(r.ExtractInt32(r.root(), unsigned_big32,
nullptr).IsInvalidArgument());
+ int64_t unsigned_big32_int64;
+ ASSERT_OK(r.ExtractInt64(r.root(),
[kudu] Git Push Summary
Repository: kudu Updated Tags: refs/tags/1.8.0-RC2 [created] cbbf7b580
svn commit: r30088 - in /dev/kudu/1.8.0-RC2: apache-kudu-1.8.0.tar.gz apache-kudu-1.8.0.tar.gz.asc apache-kudu-1.8.0.tar.gz.sha512
Author: abukor Date: Tue Oct 16 09:14:14 2018 New Revision: 30088 Log: Fixing 1.8.0 RC2 Modified: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 Modified: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz == Binary files - no diff available. Modified: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc == --- dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc (original) +++ dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc Tue Oct 16 09:14:14 2018 @@ -1,17 +1,17 @@ -BEGIN PGP SIGNATURE- -iQJGBAABCAAwFiEEOfLDizjI1U+rPiiVrM9zpE5GbYcFAlvFpiwSHGFidWtvckBh -cGFjaGUub3JnAAoJEKzPc6RORm2HPjUQAIEAcOYN8fJvBRT2u8dg1jcWRNdnWHxG -4Uy4KvJtjLZeBVeFMSNN+Ag5rl3b6oDDhif2cOEbGMK48elJQqPHZgR5Fb2aeQQS -y2SbylOl56rf3hriGqEvTA29DUPH6Jpj6axFwCowNYIytsRxhEOjhcG/Fq3OfiE5 -qTRxLWk0ePzBX/apYA6yYb7AewMkIIqnyPRhABf2aJsmNr0QcMDsheJ8xWwbgRIG -f2Orq+jo+zI46BlF+QQPoxAFcc1ibW5Y9AII2vPHUG3IdzNygRXI7Tuj3uciVh6P -Eowd0/9P6dTWptPCFLSSdp4VPBwFCIIIJYYcqGZnT9UkKuYBaNEQFUgSJQK0jYpX -XcZ07WBF/H+VjhYmzqfLnn21C6DaFmwli9h6Tie/OkEyrm9GqstHN8AmAyhGTBru -/EMjGpgSFUhhIgWc39UNveToyCVu3IOIn0qud9Kftbf7j9LlaFptp6TW0SjrUGhs -ooCZ0DPIbii4kQHonAfJS/dzJKUs49cuyzRfGu0Scyv5DfF7qUcO/YMmL2LflZNn -6/aBS7qQx3+gANia7/1ET9O1psw/awAnxHTpdZVrqcOPl9vocPNb5A1m7JSLK+FU -Ad7Xr8fS0QQsIUh+EN1tNl/NF0WNFU6l89cA4osdI+9VZYySNNgInAh7neENXBNu -tLuQIZg8Lb/2 -=teSt +iQJGBAABCAAwFiEEOfLDizjI1U+rPiiVrM9zpE5GbYcFAlvFq7YSHGFidWtvckBh +cGFjaGUub3JnAAoJEKzPc6RORm2Hg/gP/AysYLd0WkknCWbRW9qUC57j3WpvJdpS +jJYz6/hFTtTnii/Ze0EOZ5MXbnVomRDcoSAPjaGF7fT42LhRNM999WBtWNZF8rD9 +qPFc5njGpyy0fSV5KrdG7JGkWYb1rSoH1IqA/4vcWtn9UXwz4yDtRJesOWmzd5Z/ +hgo09BJSPLQYCgDvUpKPvjzwfz41YIlIBWryd53PPgvZtjfu8pU3xVmc+RZWctnY +eCgjRxbGVRQiZcoDGq6Rav9o9iGMc+siVQr8RX8Tte0jL1anHAgMotDcP+EFFu3k ++7OOcIdMdVzQn15tiEIsUTBSxxYWnk/qHLaI3ukY0YBKUuBvEifEd/xQ0YEaUa4q +TIZ8kzIug00Hswi89Uy0kg8g48bbo3QqEfI0QldGGLy7wgVA6ommUqG1O1Y1bmMB +b4PIyxABmC05kc5YnQcFBANTBhnSrpHtIf68oLlc5ATaq++zd2ScYKlFuvZD4FOp +6Ey4d//oUjCJvOTUBIlg6Od7w0iOUQ2W81/2e3gWSDPS4Qz39rOyTnhOASlI4TrB ++Y6XIm7t1bhF434bfQQYwo6ST6DgZihqVdlRtFLcTxaZHr7denSxacRrJIkVY1Bf +UyE9gKvGFHr6+8/R6kvYHNG2redd5d2ho4eUYm9Heqr/1rfYZ6qHF4mS26Jj0F89 +eK8q74JoZVfj +=9HpV -END PGP SIGNATURE- Modified: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 == --- dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 (original) +++ dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 Tue Oct 16 09:14:14 2018 @@ -1 +1 @@ -48d4b86df7341b8a0ea78c23f6049381cd80e2cf49eb3f1df4a67993844439112f7807b8baac901b13a0e4b7138adff93728829c69e3fd8e6d6c1098a1c19549 apache-kudu-1.8.0.tar.gz +68e459a7e071e02d1c39b340fb902bb4b2487d0032fa895de19111f585ebc71ae7bd17269919d1ab445028b1e0043ed9d78af04c154268d7f7a0242c0a8b860f apache-kudu-1.8.0.tar.gz
[kudu] Git Push Summary
Repository: kudu Updated Tags: refs/tags/1.8.0-RC2 [deleted] ac9e22e33
svn commit: r30087 - in /dev/kudu: 1.8.0-RC1/ 1.8.0-RC2/ 1.8.0-RC2/apache-kudu-1.8.0.tar.gz 1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc 1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512
Author: abukor Date: Tue Oct 16 08:53:25 2018 New Revision: 30087 Log: Adding Kudu 1.8.0 RC2 and removing RC1 Added: dev/kudu/1.8.0-RC2/ dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz (with props) dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 Removed: dev/kudu/1.8.0-RC1/ Added: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz == Binary file - no diff available. Propchange: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz -- svn:mime-type = application/octet-stream Added: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc == --- dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc (added) +++ dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.asc Tue Oct 16 08:53:25 2018 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJGBAABCAAwFiEEOfLDizjI1U+rPiiVrM9zpE5GbYcFAlvFpiwSHGFidWtvckBh +cGFjaGUub3JnAAoJEKzPc6RORm2HPjUQAIEAcOYN8fJvBRT2u8dg1jcWRNdnWHxG +4Uy4KvJtjLZeBVeFMSNN+Ag5rl3b6oDDhif2cOEbGMK48elJQqPHZgR5Fb2aeQQS +y2SbylOl56rf3hriGqEvTA29DUPH6Jpj6axFwCowNYIytsRxhEOjhcG/Fq3OfiE5 +qTRxLWk0ePzBX/apYA6yYb7AewMkIIqnyPRhABf2aJsmNr0QcMDsheJ8xWwbgRIG +f2Orq+jo+zI46BlF+QQPoxAFcc1ibW5Y9AII2vPHUG3IdzNygRXI7Tuj3uciVh6P +Eowd0/9P6dTWptPCFLSSdp4VPBwFCIIIJYYcqGZnT9UkKuYBaNEQFUgSJQK0jYpX +XcZ07WBF/H+VjhYmzqfLnn21C6DaFmwli9h6Tie/OkEyrm9GqstHN8AmAyhGTBru +/EMjGpgSFUhhIgWc39UNveToyCVu3IOIn0qud9Kftbf7j9LlaFptp6TW0SjrUGhs +ooCZ0DPIbii4kQHonAfJS/dzJKUs49cuyzRfGu0Scyv5DfF7qUcO/YMmL2LflZNn +6/aBS7qQx3+gANia7/1ET9O1psw/awAnxHTpdZVrqcOPl9vocPNb5A1m7JSLK+FU +Ad7Xr8fS0QQsIUh+EN1tNl/NF0WNFU6l89cA4osdI+9VZYySNNgInAh7neENXBNu +tLuQIZg8Lb/2 +=teSt +-END PGP SIGNATURE- Added: dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 == --- dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 (added) +++ dev/kudu/1.8.0-RC2/apache-kudu-1.8.0.tar.gz.sha512 Tue Oct 16 08:53:25 2018 @@ -0,0 +1 @@ +48d4b86df7341b8a0ea78c23f6049381cd80e2cf49eb3f1df4a67993844439112f7807b8baac901b13a0e4b7138adff93728829c69e3fd8e6d6c1098a1c19549 apache-kudu-1.8.0.tar.gz
[kudu] Git Push Summary
Repository: kudu Updated Tags: refs/tags/1.8.0-RC2 [created] ac9e22e33
[1/3] kudu git commit: KUDU-2463 pt 2: bump MVCC safe time on Raft no-op
Repository: kudu
Updated Branches:
refs/heads/branch-1.8.x ac9e22e33 -> cbbf7b580
KUDU-2463 pt 2: bump MVCC safe time on Raft no-op
Based on the same rationale as Part 1 of this patch series, this patch
updates MVCC's safe and clean time using the no-op timestamp provided by
the leader following a successful Raft election.
There isn't an obvious reference to the tablet (to get to the MVCC
module) in Raft consensus, but there is a ReplicaTransactionFactory,
that the TabletReplica implements. I've extended this to be a more
general ConsensusRoundHandler that can be used to create transactions or
finish transactions as needed.
An invariant we are trying to uphold is that once MVCC's safe time is
adjusted, all further transactions registered with MVCC will have higher
timestamps than the safe time. With this in mind, it is critical that
the adjustment of safe time be serialized with respect to transactions.
This is the case today because safe time is only advanced by writes in
the prepare thread, on which transactions are started. To echo this,
Raft no-ops will also adjust the safe time on the prepare thread.
The following test changes are included:
- to ensure nothing terrible happens when there is a lot of election
churn (and hence, a lot of new timestamp advancement), I've tweaked
exactly_once_writes-itest to more explicitly churn elections.
Previously it attempted this with just a low timeout. I injected some
latency to make it churn a bit harder and looped the test 1000 times
in both TSAN and debug mode.
- since MvccManager::StartTransaction() will hit a CHECK failure if it
starts a transaction at a timestamp that was previously marked safe, I
added a configurable sleep at the beginning of the function to widen
the window during which safe time can be advanced, encouraging the
CHECK failure. I configured this in raft_consensus_election-itest and
looped it 1000 times in TSAN and debug mode. If no-ops _didn't_ use
the prepare thread to advance safe time, the added delay would lead to
CHECK failures.
- added a test that ensures that, on its own, a tablet will bump its
MVCC timestamps, with just its elections
- tweaked raft_consensus-itest to use more realistic timestamps, now
that MVCC's clean and safe time gets updated with the leadership no-op
This patch alone doesn't fix KUDU-2463. Rather, a later patch will
prevent scans from occuring if the MVCC safe time hasn't been advanced,
at which point this patch will reduce the window of scan unavailability.
Change-Id: Icbf812e2cb7c322fd980245cfe40c886a15a
Reviewed-on: http://gerrit.cloudera.org:8080/11427
Tested-by: Andrew Wong
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy
Reviewed-on: http://gerrit.cloudera.org:8080/11689
Reviewed-by: Grant Henke
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/b3bb51ec
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/b3bb51ec
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/b3bb51ec
Branch: refs/heads/branch-1.8.x
Commit: b3bb51ec89c7f1856aa25c1fb015ce305b28aba3
Parents: ac9e22e
Author: Andrew Wong
Authored: Tue Sep 11 12:15:02 2018 -0700
Committer: Attila Bukor
Committed: Tue Oct 16 08:40:16 2018 +
--
src/kudu/consensus/consensus-test-util.h| 6 +-
src/kudu/consensus/raft_consensus.cc| 13 ++--
src/kudu/consensus/raft_consensus.h | 50 +++-
.../exactly_once_writes-itest.cc| 16 +++-
.../integration-tests/raft_consensus-itest.cc | 82
.../raft_consensus_election-itest.cc| 2 +
.../timestamp_advancement-itest.cc | 27 +++
src/kudu/tablet/mvcc.cc | 24 --
src/kudu/tablet/mvcc.h | 3 +-
src/kudu/tablet/tablet_replica.cc | 47 ++-
src/kudu/tablet/tablet_replica.h| 15 ++--
11 files changed, 204 insertions(+), 81 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/b3bb51ec/src/kudu/consensus/consensus-test-util.h
--
diff --git a/src/kudu/consensus/consensus-test-util.h
b/src/kudu/consensus/consensus-test-util.h
index 8b6a71e..d6dc3f0 100644
--- a/src/kudu/consensus/consensus-test-util.h
+++ b/src/kudu/consensus/consensus-test-util.h
@@ -668,7 +668,7 @@ class TestDriver {
};
// A transaction factory for tests, usually this is implemented by
TabletReplica.
-class TestTransactionFactory : public ReplicaTransactionFactory {
+class TestTransactionFactory : public ConsensusRoundHandler {
public:
explicit TestTransactionFactory(log::Log* log)
: consensus_(nullptr),
@@ -681,7 +681,7 @@ class TestTransactionFactory : public
ReplicaTransactionFactory {
consensus_ =
[2/3] kudu git commit: KUDU-2463 pt 3: don't scan if MVCC hasn't moved
KUDU-2463 pt 3: don't scan if MVCC hasn't moved
In cases when a tablet bootstrap yields an MvccManager whose safe time
has not been advanced (e.g. if there are no write ops in the WAL), and
the tablet has otherwise not bumped its MVCC timestamps (e.g. if it has
not yet elected a leader), a scan (whose correctness depends on the
MvccManager to determine what transactions have been applied) will
return incorrect results.
In the same way that we prevent compactions from occuring if MVCC's
timestamps have not been moved, this patch prevents incorrect results
from being returend from a scan by returning an error that can be
retried elsewhere.
New tests are added to attempt to scan in this state, verifying that we
get an error. A couple of tests that use the mock clock are also updated
so the initial timestamp assigned to a no-op is a more organic, non-zero
timestamp.
Change-Id: Idc0f77673e1f04a34ab1f5c1930bbaa2498b39bf
Reviewed-on: http://gerrit.cloudera.org:8080/11428
Reviewed-by: Mike Percy
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/11690
Reviewed-by: Grant Henke
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/fae09bdd
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/fae09bdd
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/fae09bdd
Branch: refs/heads/branch-1.8.x
Commit: fae09bdd659fad59c4659d0860af6449ce40a990
Parents: b3bb51e
Author: Andrew Wong
Authored: Tue Sep 4 00:55:58 2018 -0700
Committer: Attila Bukor
Committed: Tue Oct 16 08:40:22 2018 +
--
src/kudu/integration-tests/consistency-itest.cc | 9 +
.../tablet_history_gc-itest.cc | 27 +--
.../timestamp_advancement-itest.cc | 176 ++-
src/kudu/mini-cluster/external_mini_cluster.cc | 6 +
src/kudu/mini-cluster/external_mini_cluster.h | 5 +
src/kudu/mini-cluster/internal_mini_cluster.cc | 7 +
src/kudu/mini-cluster/internal_mini_cluster.h | 10 +-
src/kudu/mini-cluster/mini_cluster.h| 8 +
src/kudu/tablet/mvcc.cc | 9 +
src/kudu/tablet/mvcc.h | 5 +
src/kudu/tools/kudu-ts-cli-test.cc | 20 ++-
src/kudu/tserver/tablet_service.cc | 11 ++
12 files changed, 231 insertions(+), 62 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/fae09bdd/src/kudu/integration-tests/consistency-itest.cc
--
diff --git a/src/kudu/integration-tests/consistency-itest.cc
b/src/kudu/integration-tests/consistency-itest.cc
index a0a2dc9..81231cd 100644
--- a/src/kudu/integration-tests/consistency-itest.cc
+++ b/src/kudu/integration-tests/consistency-itest.cc
@@ -113,6 +113,15 @@ class ConsistencyITest : public MiniClusterITestBase {
virtual void SetUp() override {
MiniClusterITestBase::SetUp();
StartCluster(num_tablet_servers_);
+
+// Since we're using mock NTP rather than the hybrid clock, it's possible
+// that the first timestamp assigned to a tablet message is the initial
+// timestamp (0). For correctness of scans, it is illegal to scan in this
+// state. As such, we bump the clock up front so when we create tablets,
+// they start out with more natural, non-0 values.
+for (int i = 0; i < num_tablet_servers_; i++) {
+ cluster_->mini_tablet_server(i)->server()->clock()->Now();
+}
}
void ScannerThread(KuduClient::ReplicaSelection selection,
http://git-wip-us.apache.org/repos/asf/kudu/blob/fae09bdd/src/kudu/integration-tests/tablet_history_gc-itest.cc
--
diff --git a/src/kudu/integration-tests/tablet_history_gc-itest.cc
b/src/kudu/integration-tests/tablet_history_gc-itest.cc
index e2ec09b..40b8ad5 100644
--- a/src/kudu/integration-tests/tablet_history_gc-itest.cc
+++ b/src/kudu/integration-tests/tablet_history_gc-itest.cc
@@ -523,6 +523,23 @@ TEST_F(RandomizedTabletHistoryGcITest,
TestRandomHistoryGCWorkload) {
FLAGS_scanner_ttl_ms = 1000 * 60 * 60 * 24;
StartCluster(1); // Start InternalMiniCluster with a single tablet server.
+ // Since we're using mock NTP rather than the hybrid clock, it's possible
+ // that if we created a tablet now, the first timestamp assigned to a tablet
+ // message would be the initial timestamp (0). For correctness of scans, it
+ // is illegal to scan in this state. As such, we bump the clock up front so
+ // when we create tablets, they start out with more natural, non-0 values.
+ MiniTabletServer* mts = cluster_->mini_tablet_server(0);
+
+ // Directly access the tserver so we can control compaction and the clock.
+ TabletServer* ts = mts->server();
+ clock_ = down_cast(ts->clock());
+
+ // Set initial clock time to
kudu git commit: [tools] Add a describe table tool
Repository: kudu
Updated Branches:
refs/heads/master 1197eb01a -> 614b446e1
[tools] Add a describe table tool
This adds a very basic tool to describe a table, printing out its
schema, its partition schema, the current range partitions, and the
replication factor. It eschews completeness for brevity, leaving out,
for example, column storage attributes, since they clutter the output
and make it hard to read. My use case for this tool is as an easy way a
user can gather the fundamental info about a table and show it to
someone else. It should be easy to copy and paste, usually fit on one
line in a terminal window, and not depend on fancy formatting or a
fixed-width font to display correctly.
Sample output:
$ kudu table describe localhost:7053
default.loadgen_auto_f6d439ab21e4408486786289297f7db0
TABLE default.loadgen_auto_f6d439ab21e4408486786289297f7db0 (
key INT64 NOT NULL,
int_val INT32 NULLABLE,
string_val STRING NULLABLE,
PRIMARY KEY (key)
)
HASH (key) PARTITIONS 2,
RANGE (key) (
PARTITION UNBOUNDED
)
REPLICAS 1
Getting this "SQL-ish" look required changing how schemas and column
schemas are stringified, so there are some knock-on changes in various
tests as well.
I also reorganized the order of `kudu table` commands so they are in
alphabetical order.
Change-Id: Ib02f2a94cf11d0e60b32cd85be920ca9b99dd977
Reviewed-on: http://gerrit.cloudera.org:8080/11665
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/614b446e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/614b446e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/614b446e
Branch: refs/heads/master
Commit: 614b446e1f9291189657b265830d05d0395c5d1c
Parents: 1197eb0
Author: Will Berkeley
Authored: Thu Oct 11 11:30:36 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 16 05:39:39 2018 +
--
src/kudu/client/client-test.cc | 2 +-
src/kudu/client/client-unittest.cc | 46 +-
src/kudu/client/client.h | 11 ++-
src/kudu/client/schema.cc| 2 +-
src/kudu/common/partial_row-test.cc | 2 +-
src/kudu/common/row_changelist-test.cc | 4 +-
src/kudu/common/row_operations-test.cc | 16 ++--
src/kudu/common/schema-test.cc | 98 +++--
src/kudu/common/schema.cc| 27 +++---
src/kudu/tablet/tablet_bootstrap-test.cc | 2 +-
src/kudu/tools/CMakeLists.txt| 2 +-
src/kudu/tools/kudu-admin-test.cc| 121 ++
src/kudu/tools/kudu-tool-test.cc | 6 +-
src/kudu/tools/tool_action_common.cc | 46 +-
src/kudu/tools/tool_action_common.h | 8 +-
src/kudu/tools/tool_action_table.cc | 90 +++
src/kudu/tserver/tablet_server-test.cc | 22 ++---
17 files changed, 375 insertions(+), 130 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/614b446e/src/kudu/client/client-test.cc
--
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index 1322452..dc7e7b8 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -3618,7 +3618,7 @@ TEST_F(ClientTest, TestWriteWithBadSchema) {
unique_ptr error = GetSingleErrorFromSession(session.get());
ASSERT_TRUE(error->status().IsInvalidArgument());
ASSERT_STR_CONTAINS(error->status().ToString(),
-"Client provided column int_val[int32 NOT NULL] "
+"Client provided column int_val INT32 NOT NULL "
"not present in tablet");
ASSERT_EQ(error->failed_op().ToString(),
R"(INSERT int32 key=12345, int32 int_val=12345, string
string_val="x")");
http://git-wip-us.apache.org/repos/asf/kudu/blob/614b446e/src/kudu/client/client-unittest.cc
--
diff --git a/src/kudu/client/client-unittest.cc
b/src/kudu/client/client-unittest.cc
index d2c4866..f1b2fea 100644
--- a/src/kudu/client/client-unittest.cc
+++ b/src/kudu/client/client-unittest.cc
@@ -238,18 +238,18 @@ TEST(ClientUnitTest, TestKuduSchemaToString) {
->Default(KuduValue::FromInt(12345));
ASSERT_OK(b1.Build());
- string schema_str_1 = "Schema [\n"
-"\tprimary key (key),\n"
-"\tkey[int32 NOT NULL],\n"
-"\tint_val[int32 NOT NULL],\n"
-"\tstring_val[string NULLABLE],\n"
-"\tnon_null_with_default[int32 NOT NULL]\n"
-"]";
+ string schema_str_1 = "(\n"
+"key INT32 NOT NULL,\n"
+"int_val INT32 NOT NULL,\n"
+"
[3/3] kudu git commit: docs: update release note for KUDU-2463
docs: update release note for KUDU-2463 Change-Id: Id8dce61da14f67e39f6573fa42ec54809f3ceb19 Reviewed-on: http://gerrit.cloudera.org:8080/11691 Reviewed-by: Grant Henke Reviewed-by: Mike Percy Tested-by: Andrew Wong Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/cbbf7b58 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/cbbf7b58 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/cbbf7b58 Branch: refs/heads/branch-1.8.x Commit: cbbf7b580c4ab4fdf6621e4ee5ab1ddc5f03cb4e Parents: fae09bd Author: Andrew Wong Authored: Mon Oct 15 14:53:13 2018 -0700 Committer: Attila Bukor Committed: Tue Oct 16 08:40:28 2018 + -- docs/release_notes.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/cbbf7b58/docs/release_notes.adoc -- diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc index 0bfbec5..f02233f 100644 --- a/docs/release_notes.adoc +++ b/docs/release_notes.adoc @@ -230,8 +230,8 @@ crash the tablet server (see link:https://issues.apache.org/jira/browse/KUDU-2293[KUDU-2293]). -- Reduced the likelihood of seeing a bug in which incorrect results would be returned in - scans following a server restart (see +- Fixed a bug in which incorrect results would be returned in scans following a + server restart (see link:https://issues.apache.org/jira/browse/KUDU-2463[KUDU-2463]). - Fixed a bug causing a tablet server crash when a write batch request from a client
