kudu git commit: Fix RAT warnings
Repository: kudu Updated Branches: refs/heads/branch-1.8.x 6eb17c2d2 -> d79f80eb0 Fix RAT warnings Change-Id: If18dc7321ba0294d9d640064f25e5b3d3bca1914 Reviewed-on: http://gerrit.cloudera.org:8080/11632 Reviewed-by: Grant Henke Tested-by: Grant Henke (cherry picked from commit 88edfc36a888d53445b0e24159a10bf7de8b5213) Reviewed-on: http://gerrit.cloudera.org:8080/11631 Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d79f80eb Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d79f80eb Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d79f80eb Branch: refs/heads/branch-1.8.x Commit: d79f80eb0d71091855a587c1f6a5cdf8d374 Parents: 6eb17c2 Author: Attila Bukor Authored: Wed Oct 10 00:04:27 2018 +0200 Committer: Grant Henke Committed: Wed Oct 10 01:53:00 2018 + -- build-support/release/rat_exclude_files.txt| 6 +- .../apache/kudu/examples/KuduCollectlExample.java | 17 + .../apache/kudu/flume/sink/KuduSinkTestUtil.java | 17 + ...RegexpKuduOperationsProducerParseErrorTest.java | 17 + .../org/apache/kudu/spark/kudu/SparkUtil.scala | 17 + 5 files changed, 73 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/d79f80eb/build-support/release/rat_exclude_files.txt -- diff --git a/build-support/release/rat_exclude_files.txt b/build-support/release/rat_exclude_files.txt index 8680003..28a9132 100644 --- a/build-support/release/rat_exclude_files.txt +++ b/build-support/release/rat_exclude_files.txt @@ -16,7 +16,9 @@ build-support/iwyu/mappings/boost-all-private.imp build-support/iwyu/mappings/boost-all.imp build-support/release/rat_exclude_files.txt docs/support/doxygen/client_api.footer.in -java/kudu-flume-sink/src/test/avro/testAvroKuduOperationsProducer.avsc +examples/README.md +examples/python/graphite-kudu/setup.cfg +java/.scalafmt.conf java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java java/kudu-client/src/main/java/org/apache/kudu/client/Bytes.java @@ -29,6 +31,8 @@ java/kudu-client/src/main/java/org/apache/kudu/client/PleaseThrottleException.ja java/kudu-client/src/main/java/org/apache/kudu/client/RecoverableException.java java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java java/kudu-client/src/test/resources/flags +java/kudu-flume-sink/src/test/resources/testAvroKuduOperationsProducer.avsc +java/kudu-spark-tools/src/test/resources/TestImportExportFiles.csv python/MANIFEST.in python/README.md python/pytest.ini http://git-wip-us.apache.org/repos/asf/kudu/blob/d79f80eb/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java -- diff --git a/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java b/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java index 03ee92e..3dbfa27 100644 --- a/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java +++ b/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java @@ -1,3 +1,20 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + package org.apache.kudu.examples; import java.io.BufferedReader; http://git-wip-us.apache.org/repos/asf/kudu/blob/d79f80eb/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java -- diff --git a/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java b/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java index e24bfe0..3f5cfbf 100644 --- a/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java +++
kudu git commit: Improve logging for mini HMS and Sentry services
Repository: kudu
Updated Branches:
refs/heads/master 88edfc36a -> 61aed1a58
Improve logging for mini HMS and Sentry services
This commit adds custom logging configurations for both mini HMS and
Sentry. Both services now log warnings or errors to the console, and
info level messages to a file in the mini cluster root. This cuts down
on verbosity when both are active, and simplifies debugging by
separating full-fidelity logs.
The HMS console log is set to ERROR level and the Sentry console log is
set to the WARN level, based on the relative verbosity of the service
logs. Adjusting these levels as necessary in the future will be
straightforward.
Change-Id: Idd14cda83d56b0588667b9a3de260ffa62842adc
Reviewed-on: http://gerrit.cloudera.org:8080/11635
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao
Reviewed-by: Andrew Wong
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/61aed1a5
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/61aed1a5
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/61aed1a5
Branch: refs/heads/master
Commit: 61aed1a58d1510d65cb9a3370ffbe93431e54d58
Parents: 88edfc3
Author: Dan Burkert
Authored: Mon Oct 8 11:24:14 2018 -0700
Committer: Dan Burkert
Committed: Wed Oct 10 00:31:17 2018 +
--
src/kudu/hms/mini_hms.cc | 38 +
src/kudu/hms/mini_hms.h| 3 +++
src/kudu/sentry/mini_sentry.cc | 29 ++--
3 files changed, 64 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/61aed1a5/src/kudu/hms/mini_hms.cc
--
diff --git a/src/kudu/hms/mini_hms.cc b/src/kudu/hms/mini_hms.cc
index 079cdb1..771cc30 100644
--- a/src/kudu/hms/mini_hms.cc
+++ b/src/kudu/hms/mini_hms.cc
@@ -102,6 +102,7 @@ Status MiniHms::Start() {
RETURN_NOT_OK(CreateHiveSite());
RETURN_NOT_OK(CreateCoreSite());
+ RETURN_NOT_OK(CreateLogConfig());
// Comma-separated list of additional jars to add to the HMS classpath.
string aux_jars =
Substitute("$0/hms-plugin.jar,$1/hcatalog/share/hcatalog/*",
@@ -111,10 +112,6 @@ Status MiniHms::Start() {
string java_options =
// Ensure IPv4 is used.
"-Djava.net.preferIPv4Stack=true "
-// Make logging less verbose.
-"-Dhive.log.level=WARN "
-// Log to the console.
-"-Dhive.root.logger=console "
// Tune down the Derby deadlock timeout. The HMS's use of Derby with the
// NOTIFICATION_SEQUENCE table is prone to deadlocks, at which point Derby
// cancels a conflicting transaction after waiting out the timeout. This
@@ -321,5 +318,38 @@ Status MiniHms::CreateCoreSite() const {
JoinPathSegments(data_root_, "core-site.xml"));
}
+Status MiniHms::CreateLogConfig() const {
+ // Configure the HMS to output ERROR messages to the stderr console, and INFO
+ // and above to hms.log in the data root. The console messages have a special
+ // 'HMS' tag included to disambiguate them from other Java component logs.
The
+ // HMS automatically looks for a logging configuration named
+ // 'hive-log4j2.properties' in the configured HIVE_CONF_DIR.
+ static const string kFileTemplate = R"(
+appender.console.type = Console
+appender.console.name = console
+appender.console.target = SYSTEM_ERR
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} [HMS - %p - %t] (%F:%L) %m%n
+appender.console.filter.threshold.type = ThresholdFilter
+appender.console.filter.threshold.level = ERROR
+
+appender.file.type = File
+appender.file.name = file
+appender.file.fileName = $0
+appender.file.layout.type = PatternLayout
+appender.file.layout.pattern = %d{HH:mm:ss.SSS} [%p - %t] (%F:%L) %m%n
+
+rootLogger.level = INFO
+rootLogger.appenderRefs = console, file
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.file.ref = file
+ )";
+
+ string file_contents = Substitute(kFileTemplate,
JoinPathSegments(data_root_, "hms.log"));
+
+ return WriteStringToFile(Env::Default(),
+ file_contents,
+ JoinPathSegments(data_root_,
"hive-log4j2.properties"));
+}
} // namespace hms
} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/61aed1a5/src/kudu/hms/mini_hms.h
--
diff --git a/src/kudu/hms/mini_hms.h b/src/kudu/hms/mini_hms.h
index 6528e0b..b98fa4b 100644
--- a/src/kudu/hms/mini_hms.h
+++ b/src/kudu/hms/mini_hms.h
@@ -86,6 +86,9 @@ class MiniHms {
// Creates a core-site.xml for the mini HMS.
Status CreateCoreSite() const WARN_UNUSED_RESULT;
+ // Creates a log4j2 configuration properties file for the mini HMS.
+ Status CreateLogConfig() const
kudu git commit: Fix RAT warnings
Repository: kudu Updated Branches: refs/heads/master ab38f6825 -> 88edfc36a Fix RAT warnings Change-Id: If18dc7321ba0294d9d640064f25e5b3d3bca1914 Reviewed-on: http://gerrit.cloudera.org:8080/11632 Reviewed-by: Grant Henke Tested-by: Grant Henke Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/88edfc36 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/88edfc36 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/88edfc36 Branch: refs/heads/master Commit: 88edfc36a888d53445b0e24159a10bf7de8b5213 Parents: ab38f68 Author: Attila Bukor Authored: Wed Oct 10 00:04:27 2018 +0200 Committer: Attila Bukor Committed: Tue Oct 9 22:06:28 2018 + -- build-support/release/rat_exclude_files.txt| 6 +- .../apache/kudu/examples/KuduCollectlExample.java | 17 + .../apache/kudu/flume/sink/KuduSinkTestUtil.java | 17 + ...RegexpKuduOperationsProducerParseErrorTest.java | 17 + .../org/apache/kudu/spark/kudu/SparkUtil.scala | 17 + 5 files changed, 73 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/88edfc36/build-support/release/rat_exclude_files.txt -- diff --git a/build-support/release/rat_exclude_files.txt b/build-support/release/rat_exclude_files.txt index 8680003..28a9132 100644 --- a/build-support/release/rat_exclude_files.txt +++ b/build-support/release/rat_exclude_files.txt @@ -16,7 +16,9 @@ build-support/iwyu/mappings/boost-all-private.imp build-support/iwyu/mappings/boost-all.imp build-support/release/rat_exclude_files.txt docs/support/doxygen/client_api.footer.in -java/kudu-flume-sink/src/test/avro/testAvroKuduOperationsProducer.avsc +examples/README.md +examples/python/graphite-kudu/setup.cfg +java/.scalafmt.conf java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java java/kudu-client/src/main/java/org/apache/kudu/client/Bytes.java @@ -29,6 +31,8 @@ java/kudu-client/src/main/java/org/apache/kudu/client/PleaseThrottleException.ja java/kudu-client/src/main/java/org/apache/kudu/client/RecoverableException.java java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java java/kudu-client/src/test/resources/flags +java/kudu-flume-sink/src/test/resources/testAvroKuduOperationsProducer.avsc +java/kudu-spark-tools/src/test/resources/TestImportExportFiles.csv python/MANIFEST.in python/README.md python/pytest.ini http://git-wip-us.apache.org/repos/asf/kudu/blob/88edfc36/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java -- diff --git a/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java b/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java index 03ee92e..3dbfa27 100644 --- a/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java +++ b/examples/java/collectl/src/main/java/org/apache/kudu/examples/KuduCollectlExample.java @@ -1,3 +1,20 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + package org.apache.kudu.examples; import java.io.BufferedReader; http://git-wip-us.apache.org/repos/asf/kudu/blob/88edfc36/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java -- diff --git a/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java b/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java index e24bfe0..3f5cfbf 100644 --- a/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java +++ b/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTestUtil.java @@ -1,3 +1,20 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more
svn commit: r29981 - /release/kudu/KEYS
Author: abukor Date: Tue Oct 9 22:05:47 2018 New Revision: 29981 Log: Add signing key for Attila Bukor Modified: release/kudu/KEYS Modified: release/kudu/KEYS == --- release/kudu/KEYS (original) +++ release/kudu/KEYS Tue Oct 9 22:05:47 2018 @@ -688,3 +688,355 @@ TUSKePc/caQFccp8x+YtU54DMh9ahw2VOUMgE3K8 bBdjJ9irqp35aizoHDzFmDzZk1duzy4= =zWnO -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2016-05-18 [SC] [expires: 2020-05-19] + 39F2C38B38C8D54FAB3E2895ACCF73A44E466D87 +uid [ultimate] Attila Bukor +sig 3ACCF73A44E466D87 2018-10-08 Attila Bukor +sig 345F78BED5CAD8197 2017-09-19 Zoltan Chovan +sig 9AC3D32AEC0A4170 2018-10-08 Attila Sasvari +sig 3ACCF73A44E466D87 2016-05-18 Attila Bukor +sig 3ACCF73A44E466D87 2016-05-18 Attila Bukor +sig 0BB5F77A89A07B71 2018-10-08 Herczeg Zsolt +sig 468FD76C8C7D8A8E 2018-10-09 Gezapeti Cseh +uid [ultimate] Attila Bukor +sig 3ACCF73A44E466D87 2018-10-08 Attila Bukor +sig 345F78BED5CAD8197 2017-09-19 Zoltan Chovan +sig 9AC3D32AEC0A4170 2018-10-08 Attila Sasvari +sig 3ACCF73A44E466D87 2016-05-18 Attila Bukor +sig 0BB5F77A89A07B71 2018-10-08 Herczeg Zsolt +sig 468FD76C8C7D8A8E 2018-10-09 Gezapeti Cseh +uid [ultimate] Attila Bukor +sig 3ACCF73A44E466D87 2018-10-08 Attila Bukor +sig 9AC3D32AEC0A4170 2018-10-08 Attila Sasvari +sig 3ACCF73A44E466D87 2017-06-26 Attila Bukor +sig 0BB5F77A89A07B71 2018-10-08 Herczeg Zsolt +sig 468FD76C8C7D8A8E 2018-10-09 Gezapeti Cseh +uid [ultimate] Attila Bukor +sig 3ACCF73A44E466D87 2018-10-08 Attila Bukor +sig 9AC3D32AEC0A4170 2018-10-08 Attila Sasvari +sig 3ACCF73A44E466D87 2018-07-23 Attila Bukor +sig 0BB5F77A89A07B71 2018-10-08 Herczeg Zsolt +sig 468FD76C8C7D8A8E 2018-10-09 Gezapeti Cseh +sub rsa4096 2016-05-18 [E] [expires: 2020-05-18] +sig ACCF73A44E466D87 2016-05-18 Attila Bukor +sub rsa4096 2018-10-08 [A] [expires: 2020-05-19] +sig ACCF73A44E466D87 2018-10-08 Attila Bukor + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBFc7+dQBEADeh1AoTwNnag9+8tN8z8hYwVkEQglUi3uVSz6Wo1mkWC8tQb/C +dJxSsfWOTeAaYBxViD3Gqe5x3RpSjSIS8vGxwaiN58Zjd/VAMLZXQ0URoNqHqVdH +0W2cIA2l/CJB9dVBeF1Jtobscso4Jm8rYNPjFsigfwKpmyK+vqbKCYSADrBrZ33d +zsfn5yM5HLvMN7dI4BC80ScXYcOVrTrLfhm2rvD1MM6DxBSwTBXMEi4Xd+WY+6Xd +cwykql+rupTat+8pIWOQazstwtplqIyPYUso6sITrndc9uKoHBUDB0HaiewEoa4y +IHttAoCys9baq8dHdVDoQ7zv/IVQoQF/AYEivIjvm4QUVPl098sp9/5vmp7Yt5XK +CamzBd9ppt1GnJfASX2RpHRp3GZtphO970NMVnGleceDNU5o2fNyWxjzsh/6QxQf +Ku+5ev0lQY4zjUlILyFabE5I17L7yCQk+PGWzce4wIsBECKlN9+shfw6WxAqTuDk +Kk6XRdpefsv+6/k4cREdQPOcPNVsOVP6B9frNB0yUlaH5UyE0Ph9iVIeWqGXrIDu +Mjbaox5d3uLFbONPEs6jhY5q2nX+TUfFql8QAkc9fiawxWwvgyWQjlb3qYgMi/qw +ZIGCjZTo8Z20/jAKtHETGwOpIzhB4t9WNPHw1g5BT7ZEHs5mT/X5lldFLQARAQAB +tCVBdHRpbGEgQnVrb3IgPGF0dGlsYS5idWtvckBnbWFpbC5jb20+iQJXBBMBCgBB +AhsDBQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAhkBFiEEOfLDizjI1U+rPiiVrM9z +pE5GbYcFAlu7iFUFCQeIEgEACgkQrM9zpE5GbYdbIg//dmUCC11zgFO4KtZCOhd7 +TnLw/Nwlvr+IlkMrfJsQ6J2yTnI9x//oxS8rg79iUxFaO9RzuvjHzWDPAWWtLjvS +t0kBvMnnN2sPLJGAqRs+6caTz/ewzaZvXrCzO8MJ5vvWKvzuou1rSMeiJP0UGeXZ +1+grWH+i5XBLjK6OgJnXeJ4meSoQOTkSkGQOoAeLTOd7op/gd87mHvkrVAdZMOK+ +yQwF4fucqW5oiZoSr15SWOFrAnTaL8vpTpuMXzPZ7hNT3QTuwPflaLnywXUdJfa6 +tIX3JWSBgGTPte80reT3EQGQQL0tQ97waMPjEwOWL8WH4BFHZNSbZ3SDFPrqhQJa +N3hLitt2d3ZLP9LkHa2RAlI2jz84z4LhdCTJiZtWz7HZul3Aqu4p2p2MRgfb0b8S +fm4bvzYDArQZp+REc0IsG5ywiGYWP6dL2ivU+/sFMfxdN6Hp1SHsl2u6XuzwFBRu +wlZT0kWfDztcHBRRVi8TA/d0opoZZ/cdl6Q6mtSiJGjhrLwG9g2tmi2Qc2oGmHBy +M7sGDzZz3B/FKOXTXgNBvNkS2kT3h43E+gtgvSd2T99WkihpqN7sc5c5P+j8QVAU +J3RqEd25iA4i/zAibR/vneWgldyT2xI5r6SICJCupRugsPdPxqzlGc+ve8o7xTNt +igfxGsHhdsDNH2fPoZlxZVyJAiIEEwEKAAwFAlnBgAEFgweGH4AACgkQRfeL7Vyt +gZetYhAAiTLIlMCNLPP2SJ32r+HuD7uLfPPyb19sCvGFN+XPYvRzcFjTlgue5Io2 +GlXCRfUhmeg7Wz7UAE/9j0iQg+3oZ9U6c7Ky1lyTsnTqRqSNPIlRYKaUeLsTfAvM +9FZRMwD3mQu1Gid9aQBIggv1DyXqpsPF/gu2A1OCEndwFa02KXigT1vRV40quLIH +3aodU89QY50zKagDKU7UduNuDu3Cf6TrFYEUJz3y4eQKNaAmAT40738T3bN3rI2n +GK5D6ieR7lCzmVfE0ohxeLSXQdj6GgYGmMyGdfJXio71dWKb+7YXM2s4rXwc3aIF +sxFTvFojZjtSbJlkmMftDkAFV1z7FEjxE4seGP0Fpg6Ded5QEhwy6VJK50ps/faw +CvFROitDGPHaZYH35IToYO6zxYooEdhXPhxDEK7DL/TzDQM2d/K8dB2rJbr1Po0E +mLqTbc1C8Uukqu7XBcZi/k7KaUr0V3+Fj63lFB3s6Y1QTOuMC1uLoUoKvm2ys5Cs +fJ3mlIhpRaaMjxLcWnkMarOz+Ekl2AS5/SGr1aqINyxksZ9mmNR1WHUhTu0aaFzC +iCCsnoL4B6f7U+BZuVOqFb1sde3Xo8cbrFip8bLZUdf89VonkkXsEbF3BO6X61c1 +ANclWiwOIVbNuYUzgwtA3s10jaYX9XrdzIfzBm5rrUDVBNOP4RCJATMEEAEIAB0W +IQTK8wRXI9P4L/9QoPmaw9Mq7ApBcAUCW7tgHwAKCRCaw9Mq7ApBcI7RCADSAVhK +n0dWLye6abxSUUqZwkV8vQzwaNXUzHvf2KFSyZFy9sxPZXcCrdUIi3iU/ePyWu96 +QBY/ACgJQUAtMcf8zqBmWV8/jDwStSK7TmK3iZJztUd4tXvkqjI6r9+8LTQBJbsC +F89F6nC08qEe16Na9QYeD53d0o48Ks4IKHxnVNYEFwg6PloLlykrePAjDm+X2aMN
kudu git commit: Bump version to 1.8.0 (non-SNAPSHOT)
Repository: kudu
Updated Branches:
refs/heads/branch-1.8.x 5d8f47977 -> 6eb17c2d2
Bump version to 1.8.0 (non-SNAPSHOT)
Change-Id: If5d29434950913ed7eebe0cd7fb45043ad055abf
Reviewed-on: http://gerrit.cloudera.org:8080/11629
Reviewed-by: Grant Henke
Tested-by: Grant Henke
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/6eb17c2d
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/6eb17c2d
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/6eb17c2d
Branch: refs/heads/branch-1.8.x
Commit: 6eb17c2d2e567d2a82fdd9286cfcedb369ded4a3
Parents: 5d8f479
Author: Attila Bukor
Authored: Tue Oct 9 22:58:11 2018 +0200
Committer: Attila Bukor
Committed: Tue Oct 9 21:03:16 2018 +
--
java/gradle.properties | 2 +-
java/kudu-backup/pom.xml | 2 +-
java/kudu-client-tools/pom.xml | 2 +-
java/kudu-client/pom.xml | 2 +-
java/kudu-flume-sink/pom.xml | 2 +-
java/kudu-hive/pom.xml | 2 +-
java/kudu-jepsen/pom.xml | 2 +-
java/kudu-mapreduce/pom.xml| 2 +-
java/kudu-spark-tools/pom.xml | 2 +-
java/kudu-spark/pom.xml| 2 +-
java/pom.xml | 2 +-
version.txt| 2 +-
12 files changed, 12 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/gradle.properties
--
diff --git a/java/gradle.properties b/java/gradle.properties
index 000e26f..26646de 100755
--- a/java/gradle.properties
+++ b/java/gradle.properties
@@ -20,7 +20,7 @@
#
https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties
group = org.apache.kudu
-version = 1.8.0-SNAPSHOT
+version = 1.8.0
url = https://kudu.apache.org/
# The Maven respository used when uploading artifacts.
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-backup/pom.xml
--
diff --git a/java/kudu-backup/pom.xml b/java/kudu-backup/pom.xml
index 891960c..829c6b5 100644
--- a/java/kudu-backup/pom.xml
+++ b/java/kudu-backup/pom.xml
@@ -24,7 +24,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.8.0
kudu-backup${spark.version.label}_${scala.binary.version}
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-client-tools/pom.xml
--
diff --git a/java/kudu-client-tools/pom.xml b/java/kudu-client-tools/pom.xml
index 35a45c7..75b5914 100644
--- a/java/kudu-client-tools/pom.xml
+++ b/java/kudu-client-tools/pom.xml
@@ -23,7 +23,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.8.0
kudu-client-tools
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-client/pom.xml
--
diff --git a/java/kudu-client/pom.xml b/java/kudu-client/pom.xml
index a01965f..fb4935d 100644
--- a/java/kudu-client/pom.xml
+++ b/java/kudu-client/pom.xml
@@ -23,7 +23,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.8.0
kudu-client
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-flume-sink/pom.xml
--
diff --git a/java/kudu-flume-sink/pom.xml b/java/kudu-flume-sink/pom.xml
index 7b664fb..5c24c8a 100644
--- a/java/kudu-flume-sink/pom.xml
+++ b/java/kudu-flume-sink/pom.xml
@@ -15,7 +15,7 @@
kudu-parent
org.apache.kudu
-1.8.0-SNAPSHOT
+1.8.0
kudu-flume-sink
Kudu Flume NG Sink
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-hive/pom.xml
--
diff --git a/java/kudu-hive/pom.xml b/java/kudu-hive/pom.xml
index 33d4de0..1992621 100644
--- a/java/kudu-hive/pom.xml
+++ b/java/kudu-hive/pom.xml
@@ -22,7 +22,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.8.0
kudu-hive
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-jepsen/pom.xml
--
diff --git a/java/kudu-jepsen/pom.xml b/java/kudu-jepsen/pom.xml
index 055ce31..9edfb4e 100644
--- a/java/kudu-jepsen/pom.xml
+++ b/java/kudu-jepsen/pom.xml
@@ -20,7 +20,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.8.0
kudu-jepsen
http://git-wip-us.apache.org/repos/asf/kudu/blob/6eb17c2d/java/kudu-mapreduce/pom.xml
--
diff --git a/java/kudu-mapreduce/pom.xml
kudu git commit: Update version to 1.9.0-SNAPSHOT
Repository: kudu
Updated Branches:
refs/heads/master 5d8f47977 -> ab38f6825
Update version to 1.9.0-SNAPSHOT
Change-Id: If5ed2d27b6278b7cbb704654cfb92be6dc7adeba
Reviewed-on: http://gerrit.cloudera.org:8080/11628
Reviewed-by: Grant Henke
Tested-by: Grant Henke
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/ab38f682
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/ab38f682
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/ab38f682
Branch: refs/heads/master
Commit: ab38f6825dccf95710ba48d775aa74282dca1b84
Parents: 5d8f479
Author: Attila Bukor
Authored: Tue Oct 9 22:51:59 2018 +0200
Committer: Attila Bukor
Committed: Tue Oct 9 21:03:07 2018 +
--
java/gradle.properties | 2 +-
java/kudu-backup/pom.xml | 2 +-
java/kudu-client-tools/pom.xml | 2 +-
java/kudu-client/pom.xml | 2 +-
java/kudu-flume-sink/pom.xml | 2 +-
java/kudu-hive/pom.xml | 2 +-
java/kudu-jepsen/pom.xml | 2 +-
java/kudu-mapreduce/pom.xml| 2 +-
java/kudu-spark-tools/pom.xml | 2 +-
java/kudu-spark/pom.xml| 2 +-
java/pom.xml | 2 +-
python/setup.py| 2 +-
version.txt| 2 +-
13 files changed, 13 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/gradle.properties
--
diff --git a/java/gradle.properties b/java/gradle.properties
index 000e26f..9cc522b 100755
--- a/java/gradle.properties
+++ b/java/gradle.properties
@@ -20,7 +20,7 @@
#
https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties
group = org.apache.kudu
-version = 1.8.0-SNAPSHOT
+version = 1.9.0-SNAPSHOT
url = https://kudu.apache.org/
# The Maven respository used when uploading artifacts.
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-backup/pom.xml
--
diff --git a/java/kudu-backup/pom.xml b/java/kudu-backup/pom.xml
index 891960c..ca1f1c8 100644
--- a/java/kudu-backup/pom.xml
+++ b/java/kudu-backup/pom.xml
@@ -24,7 +24,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-backup${spark.version.label}_${scala.binary.version}
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-client-tools/pom.xml
--
diff --git a/java/kudu-client-tools/pom.xml b/java/kudu-client-tools/pom.xml
index 35a45c7..e1b6f9f 100644
--- a/java/kudu-client-tools/pom.xml
+++ b/java/kudu-client-tools/pom.xml
@@ -23,7 +23,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-client-tools
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-client/pom.xml
--
diff --git a/java/kudu-client/pom.xml b/java/kudu-client/pom.xml
index a01965f..3b3babe 100644
--- a/java/kudu-client/pom.xml
+++ b/java/kudu-client/pom.xml
@@ -23,7 +23,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-client
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-flume-sink/pom.xml
--
diff --git a/java/kudu-flume-sink/pom.xml b/java/kudu-flume-sink/pom.xml
index 7b664fb..c2eaef0 100644
--- a/java/kudu-flume-sink/pom.xml
+++ b/java/kudu-flume-sink/pom.xml
@@ -15,7 +15,7 @@
kudu-parent
org.apache.kudu
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-flume-sink
Kudu Flume NG Sink
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-hive/pom.xml
--
diff --git a/java/kudu-hive/pom.xml b/java/kudu-hive/pom.xml
index 33d4de0..7239c7e 100644
--- a/java/kudu-hive/pom.xml
+++ b/java/kudu-hive/pom.xml
@@ -22,7 +22,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-hive
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-jepsen/pom.xml
--
diff --git a/java/kudu-jepsen/pom.xml b/java/kudu-jepsen/pom.xml
index 055ce31..535ffbc 100644
--- a/java/kudu-jepsen/pom.xml
+++ b/java/kudu-jepsen/pom.xml
@@ -20,7 +20,7 @@
org.apache.kudu
kudu-parent
-1.8.0-SNAPSHOT
+1.9.0-SNAPSHOT
kudu-jepsen
http://git-wip-us.apache.org/repos/asf/kudu/blob/ab38f682/java/kudu-mapreduce/pom.xml
[kudu] Git Push Summary
Repository: kudu Updated Branches: refs/heads/branch-1.8.x [created] 5d8f47977
kudu git commit: [docs] Fix placement of decommission docs in rebalancing section
Repository: kudu Updated Branches: refs/heads/master b193e3786 -> 5d8f47977 [docs] Fix placement of decommission docs in rebalancing section Change-Id: I8f4f4fa755a34e21ef26884e998d0e17ba512824 Reviewed-on: http://gerrit.cloudera.org:8080/11627 Tested-by: Will Berkeley Reviewed-by: Andrew Wong Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/5d8f4797 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/5d8f4797 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/5d8f4797 Branch: refs/heads/master Commit: 5d8f4797786c8afbdc6e93dd71eb8d73657e7591 Parents: b193e37 Author: Will Berkeley Authored: Tue Oct 9 10:53:31 2018 -0700 Committer: Will Berkeley Committed: Tue Oct 9 17:55:29 2018 + -- docs/administration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/5d8f4797/docs/administration.adoc -- diff --git a/docs/administration.adoc b/docs/administration.adoc index 1c3ab20..76a9dd9 100644 --- a/docs/administration.adoc +++ b/docs/administration.adoc @@ -1251,7 +1251,7 @@ factor one tables is not supported, it will rebalance all the other tables and the cluster as if those singly-replicated tables did not exist. [[tablet_server_decommissioning]] -.Decommissioning or Permanently Removing a Tablet Server From a Cluster +=== Decommissioning or Permanently Removing a Tablet Server From a Cluster Kudu does not currently have an automated way to remove a tablet server from a cluster permanently. Instead, use the following steps:
[1/2] kudu git commit: [webui] KUDU-844 and other /tablet-rowsetlayout-svg improvements
Repository: kudu
Updated Branches:
refs/heads/master 9ec8d28db -> b193e3786
[webui] KUDU-844 and other /tablet-rowsetlayout-svg improvements
This patch addresses KUDU-844, an issue where rowsets unavailable for
compaction weren't shown in the rowset SVG layout. These rowsets aren't
displayed because the rowset SVG is built from compaction policy output,
and the compaction policy obviously does not consider rowsets that
aren't available for compaction. There are two kinds of rowsets that
will be excluded: the memrowset, which doesn't belong in the SVG output,
and rowsets currently being compacted. The latter are tricky to
integrate into the SVG because there is a race with the compaction
finishing, and after a compaction the rowset may no longer be
relevant (merge compaction), or its bounds may have changed (major delta
compaction). Working around this problem would be a lot of effort, just
to add dubiously useful information to the layout diagram. Instead, I
opted just to disclose the issue: there's a note saying how many rowsets
are currently being compacted included under the layout diagram.
I also made a couple of quality of life improvements to the rowset
layout page.
1. I removed the size label from the rowsets rectangles in the SVG. They
didn't add so much individually, but they did make it really hard to
see what was going in once there were a lot of rowsets.
2. I added a table with the 5-number summary of rowset sizes, to make up
for taking away the individual sizes in the diagram. I chose the
summary because in my experience it's the best way to get a quick
idea what a distribution looks like (better than, e.g., the mean and
standard deviation.)
Here's the original way a busy layout diagram looks with the sizes:
https://github.com/wdberkeley/kudu/blob/svg_screenshots/layout.png
And here's how it looks without them:
https://github.com/wdberkeley/kudu/blob/svg_screenshots/layout_no_text.png
Here's an example summary table:
https://github.com/wdberkeley/kudu/blob/svg_screenshots/new_layout_page.png
Finally, I added some pairs that were missing from the /tablets
template.
Change-Id: I7475f14093a187fde1329546b10c314da49fe08a
Reviewed-on: http://gerrit.cloudera.org:8080/11613
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/91faf2a1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/91faf2a1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/91faf2a1
Branch: refs/heads/master
Commit: 91faf2a1205e32e9d8293e263300718499bdf575
Parents: 9ec8d28
Author: Will Berkeley
Authored: Sun Oct 7 22:49:58 2018 -0700
Committer: Andrew Wong
Committed: Tue Oct 9 17:37:53 2018 +
--
src/kudu/tablet/svg_dump.cc | 4 ---
src/kudu/tablet/tablet.cc | 70 ++--
www/tablets.mustache| 4 +--
3 files changed, 69 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/91faf2a1/src/kudu/tablet/svg_dump.cc
--
diff --git a/src/kudu/tablet/svg_dump.cc b/src/kudu/tablet/svg_dump.cc
index 9c88655..93a3cc2 100644
--- a/src/kudu/tablet/svg_dump.cc
+++ b/src/kudu/tablet/svg_dump.cc
@@ -136,10 +136,6 @@ void DumpSVG(const vector& candidates,
R"()",
x, y, width, kRowHeight, color)
<< endl;
- out << Substitute(R"+($4MB)+",
-x, y + kRowHeight, width, kRowHeight, cand->size_mb())
- << endl;
}
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/91faf2a1/src/kudu/tablet/tablet.cc
--
diff --git a/src/kudu/tablet/tablet.cc b/src/kudu/tablet/tablet.cc
index 05a024f..0a7a59a 100644
--- a/src/kudu/tablet/tablet.cc
+++ b/src/kudu/tablet/tablet.cc
@@ -172,6 +172,7 @@ using kudu::MaintenanceManager;
using kudu::clock::HybridClock;
using kudu::fs::IOContext;
using kudu::log::LogAnchorRegistry;
+using std::endl;
using std::ostream;
using std::pair;
using std::shared_ptr;
@@ -2299,13 +2300,76 @@ void Tablet::PrintRSLayout(ostream* o) {
RowSetInfo::CollectOrdered(*rowsets_copy, , );
DumpCompactionSVG(min, picked, o, /*print_xml_header=*/false);
- *o << "Compaction policy log" << std::endl;
+ // Compaction policy ignores rowsets unavailable for compaction. This is
good,
+ // except it causes the SVG to be potentially missing rowsets. It's hard to
+ // take these presently-compacting rowsets into account because we are racing
+ // against the compaction finishing, and at the end of the compaction the
+ // rowsets might no longer exist (merge compaction) or their bounds may have
+ // changed (major delta compaction). So, let's just disclose how many of
[2/2] kudu git commit: [tests] location assignment for ExternaMiniCluster
[tests] location assignment for ExternaMiniCluster
Introduced location assignment rules for tablet servers running as
part of ExternalMiniCluster. With this changelist, it's possible
to have tablet servers of ExternalMiniCluster to be spread among
locations in accordance with simple number-of-servers-at-location
mapping rules.
For example, the following is a self-descriptive example of such
mapping rules set to spread six tablet servers in a cluster among
three locations:
L0:1 L1:2 L2:3
Added an integration test to verify the location assignment
rules for ExternalMiniCluster work as intended.
Change-Id: I63309804cf2fdd8a620b50abdd7133af6a033c30
Reviewed-on: http://gerrit.cloudera.org:8080/11606
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/b193e378
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/b193e378
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/b193e378
Branch: refs/heads/master
Commit: b193e3786861686590b6bf67bc507d8d4d18d8e3
Parents: 91faf2a
Author: Alexey Serbin
Authored: Fri Oct 5 22:12:41 2018 -0700
Committer: Alexey Serbin
Committed: Tue Oct 9 17:42:50 2018 +
--
src/kudu/integration-tests/CMakeLists.txt | 2 +
.../integration-tests/cluster_itest_util.cc | 1 +
src/kudu/integration-tests/cluster_itest_util.h | 1 +
.../scripts/assign-location.py | 244 +++
.../ts_location_assignment-itest.cc | 132 ++
src/kudu/master/ts_descriptor.cc| 20 +-
src/kudu/mini-cluster/external_mini_cluster.cc | 24 +-
src/kudu/mini-cluster/external_mini_cluster.h | 10 +
8 files changed, 428 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/b193e378/src/kudu/integration-tests/CMakeLists.txt
--
diff --git a/src/kudu/integration-tests/CMakeLists.txt
b/src/kudu/integration-tests/CMakeLists.txt
index 512a454..7ecb697 100644
--- a/src/kudu/integration-tests/CMakeLists.txt
+++ b/src/kudu/integration-tests/CMakeLists.txt
@@ -114,6 +114,8 @@ ADD_KUDU_TEST(tombstoned_voting-imc-itest)
ADD_KUDU_TEST(tombstoned_voting-itest)
ADD_KUDU_TEST(tombstoned_voting-stress-test RUN_SERIAL true)
ADD_KUDU_TEST(token_signer-itest)
+ADD_KUDU_TEST(ts_location_assignment-itest
+ DATA_FILES scripts/assign-location.py)
ADD_KUDU_TEST(ts_recovery-itest PROCESSORS 4)
ADD_KUDU_TEST(ts_tablet_manager-itest)
ADD_KUDU_TEST(update_scan_delta_compact-test RUN_SERIAL true)
http://git-wip-us.apache.org/repos/asf/kudu/blob/b193e378/src/kudu/integration-tests/cluster_itest_util.cc
--
diff --git a/src/kudu/integration-tests/cluster_itest_util.cc
b/src/kudu/integration-tests/cluster_itest_util.cc
index 7231094..186cf01 100644
--- a/src/kudu/integration-tests/cluster_itest_util.cc
+++ b/src/kudu/integration-tests/cluster_itest_util.cc
@@ -308,6 +308,7 @@ Status CreateTabletServerMap(const
shared_ptr& master_proxy,
unique_ptr peer(new TServerDetails);
peer->instance_id.CopyFrom(entry.instance_id());
peer->registration.CopyFrom(entry.registration());
+peer->location = entry.location();
CreateTsClientProxies(addresses[0],
messenger,
http://git-wip-us.apache.org/repos/asf/kudu/blob/b193e378/src/kudu/integration-tests/cluster_itest_util.h
--
diff --git a/src/kudu/integration-tests/cluster_itest_util.h
b/src/kudu/integration-tests/cluster_itest_util.h
index c5ba53a..1768880 100644
--- a/src/kudu/integration-tests/cluster_itest_util.h
+++ b/src/kudu/integration-tests/cluster_itest_util.h
@@ -79,6 +79,7 @@ struct TServerDetails {
std::unique_ptr tserver_admin_proxy;
std::unique_ptr consensus_proxy;
std::unique_ptr generic_proxy;
+ std::string location;
// Convenience function to get the UUID from the instance_id struct.
const std::string& uuid() const;
http://git-wip-us.apache.org/repos/asf/kudu/blob/b193e378/src/kudu/integration-tests/scripts/assign-location.py
--
diff --git a/src/kudu/integration-tests/scripts/assign-location.py
b/src/kudu/integration-tests/scripts/assign-location.py
new file mode 100644
index 000..5714772
--- /dev/null
+++ b/src/kudu/integration-tests/scripts/assign-location.py
@@ -0,0 +1,244 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the
[4/4] kudu git commit: KUDU-2324 Add gflags to disable individual maintenance ops
KUDU-2324 Add gflags to disable individual maintenance ops
This patch adds one flag per maintenance op that allows the user to
individually disable each of the seven types of maintenance op. This
results in six new flags:
-enable_flush_deltamemstores
-enable_flush_memrowset
-enable_log_gc
-enable_major_delta_compaction
-enable_minor_delta_compaction
-enable_rowset_compaction
These flags do not stop the ops from being registered; instead, they
keep them from being runnable. This allows the flags to be changed at
runtime.
There was already a flag -enable_undo_delta_block_gc controlling whether
undo delta block gc was enabled; however, it prevented the
maintenance op from even being registered. The implementation of the
flag has been changed to match the others. The effect is the same,
except now the flag can be changed at runtime.
Change-Id: If4823c067883897718cc225ef85a0aaf67f1df38
Reviewed-on: http://gerrit.cloudera.org:8080/11609
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/9ec8d28d
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/9ec8d28d
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/9ec8d28d
Branch: refs/heads/master
Commit: 9ec8d28dbed301c3d0bd3d9daab183d802f6a58d
Parents: 0c91532
Author: Will Berkeley
Authored: Sun Oct 7 11:07:21 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 9 16:44:10 2018 +
--
src/kudu/tablet/tablet.cc| 15 ++-
src/kudu/tablet/tablet_mm_ops.cc | 59 +++
src/kudu/tablet/tablet_replica_mm_ops.cc | 44
3 files changed, 106 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/9ec8d28d/src/kudu/tablet/tablet.cc
--
diff --git a/src/kudu/tablet/tablet.cc b/src/kudu/tablet/tablet.cc
index ac18d41..05a024f 100644
--- a/src/kudu/tablet/tablet.cc
+++ b/src/kudu/tablet/tablet.cc
@@ -157,13 +157,6 @@ DEFINE_int32(max_encoded_key_size_bytes, 16 * 1024,
"result in an error.");
TAG_FLAG(max_encoded_key_size_bytes, unsafe);
-DEFINE_bool(enable_undo_delta_block_gc, true,
-"Whether to enable undo delta block garbage collection. "
-"This only affects the undo delta block deletion background task, and "
-"doesn't control whether compactions delete ancient history. "
-"To change what is considered ancient history use
--tablet_history_max_age_sec");
-TAG_FLAG(enable_undo_delta_block_gc, evolving);
-
METRIC_DEFINE_entity(tablet);
METRIC_DEFINE_gauge_size(tablet, memrowset_size, "MemRowSet Memory Usage",
kudu::MetricUnit::kBytes,
@@ -1422,11 +1415,9 @@ void Tablet::RegisterMaintenanceOps(MaintenanceManager*
maint_mgr) {
maint_mgr->RegisterOp(major_delta_compact_op.get());
maintenance_ops.push_back(major_delta_compact_op.release());
- if (FLAGS_enable_undo_delta_block_gc) {
-gscoped_ptr undo_delta_block_gc_op(new
UndoDeltaBlockGCOp(this));
-maint_mgr->RegisterOp(undo_delta_block_gc_op.get());
-maintenance_ops.push_back(undo_delta_block_gc_op.release());
- }
+ gscoped_ptr undo_delta_block_gc_op(new
UndoDeltaBlockGCOp(this));
+ maint_mgr->RegisterOp(undo_delta_block_gc_op.get());
+ maintenance_ops.push_back(undo_delta_block_gc_op.release());
std::lock_guard l(state_lock_);
maintenance_ops_.swap(maintenance_ops);
http://git-wip-us.apache.org/repos/asf/kudu/blob/9ec8d28d/src/kudu/tablet/tablet_mm_ops.cc
--
diff --git a/src/kudu/tablet/tablet_mm_ops.cc b/src/kudu/tablet/tablet_mm_ops.cc
index 3af1042..c71bc6e 100644
--- a/src/kudu/tablet/tablet_mm_ops.cc
+++ b/src/kudu/tablet/tablet_mm_ops.cc
@@ -18,6 +18,7 @@
#include "kudu/tablet/tablet_mm_ops.h"
#include
+#include
#include
#include
@@ -41,6 +42,36 @@ DEFINE_int32(undo_delta_block_gc_init_budget_millis, 1000,
TAG_FLAG(undo_delta_block_gc_init_budget_millis, evolving);
TAG_FLAG(undo_delta_block_gc_init_budget_millis, advanced);
+DEFINE_bool(enable_major_delta_compaction, true,
+"Whether to enable major delta compaction. Disabling major delta "
+"compaction may worsen performance and increase disk space usage for "
+"workloads involving updates and deletes.");
+TAG_FLAG(enable_major_delta_compaction, runtime);
+TAG_FLAG(enable_major_delta_compaction, unsafe);
+
+DEFINE_bool(enable_minor_delta_compaction, true,
+"Whether to enable minor delta compaction. Disabling minor delta "
+"compaction may worsen performance and increase disk space usage for "
+"workloads involving updates and deletes.");
+TAG_FLAG(enable_minor_delta_compaction, runtime);
+TAG_FLAG(enable_minor_delta_compaction, unsafe);
+
[3/4] kudu git commit: [compaction] Small improvements to compaction policy tests
[compaction] Small improvements to compaction policy tests
To learn more about compaction policy, I spent some time reading over
compaction_policy-test.cc, and in doing so I made a bunch of small
improvements. I also added one test covering a facet of the policy
that's pretty important but isn't obviously covered in the design docs
or the existing tests.
There are no functional non-test changes in this patch. I added
scaled-down versions of a couple of slow tests that can run all the
time, and made one slow test only run when slow tests are allowed.
Change-Id: I8f7a0e2c69f301d7fc7ca2fac657569ea240d4e3
Reviewed-on: http://gerrit.cloudera.org:8080/11605
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/0c91532e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/0c91532e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/0c91532e
Branch: refs/heads/master
Commit: 0c91532e742df3b8f09d94ede5d7cde701b3857c
Parents: 77ead35
Author: Will Berkeley
Authored: Fri Oct 5 20:41:19 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 9 16:44:00 2018 +
--
src/kudu/tablet/compaction_policy-test.cc | 196 +
1 file changed, 131 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/0c91532e/src/kudu/tablet/compaction_policy-test.cc
--
diff --git a/src/kudu/tablet/compaction_policy-test.cc
b/src/kudu/tablet/compaction_policy-test.cc
index ddf917f..4277f9d 100644
--- a/src/kudu/tablet/compaction_policy-test.cc
+++ b/src/kudu/tablet/compaction_policy-test.cc
@@ -60,124 +60,188 @@ class TestCompactionPolicy : public KuduTest {
BudgetedCompactionPolicy policy(size_budget_mb);
-ASSERT_OK(policy.PickRowSets(tree, picked, quality, nullptr));
+ASSERT_OK(policy.PickRowSets(tree, picked, quality, /*log=*/nullptr));
}
};
// Simple test for budgeted compaction: with three rowsets which
-// mostly overlap, and an high budget, they should all be selected.
+// mostly overlap, and a high budget, they should all be selected.
TEST_F(TestCompactionPolicy, TestBudgetedSelection) {
- RowSetVector vec = {
+ /*
+ * [C -- c]
+ * [B - a]
+ * [A --- b]
+ */
+ const RowSetVector rowsets = {
std::make_shared("C", "c"),
std::make_shared("B", "a"),
std::make_shared("A", "b")
};
- const int kBudgetMb = 1000; // enough to select all
+ constexpr auto kBudgetMb = 1000; // Enough to select all rowsets.
unordered_set picked;
- double quality = 0;
- NO_FATALS(RunTestCase(vec, kBudgetMb, , ));
- ASSERT_EQ(3, picked.size());
+ double quality = 0.0;
+ NO_FATALS(RunTestCase(rowsets, kBudgetMb, , ));
+ ASSERT_EQ(rowsets.size(), picked.size());
+ // Since all three rowsets are picked and each covers almost the entire key
+ // range, the sum of widths is about 3 while the union width is 1, so the
+ // quality score should be between 1 and 2.
ASSERT_GE(quality, 1.0);
+ ASSERT_LE(quality, 2.0);
}
// Test for the case when we have many rowsets, but none of them
// overlap at all. This is likely to occur in workloads where the
// primary key is always increasing (such as a timestamp).
-TEST_F(TestCompactionPolicy, TestNonOverlappingRowsets) {
- if (!AllowSlowTests()) {
-LOG(INFO) << "Skipped";
-return;
- }
- RowSetVector vec;
- for (int i = 0; i < 1; i++) {
-vec.emplace_back(new MockDiskRowSet(
+TEST_F(TestCompactionPolicy, TestNonOverlappingRowSets) {
+ /* NB: Zero-padding of string keys omitted to save space.
+ *
+ * [0 - 1] [2 - 3] ... [198 - 199]
+ */
+ const auto kNumRowSets = AllowSlowTests() ? 1 : 100;
+ RowSetVector rowsets;
+ for (auto i = 0; i < kNumRowSets; i++) {
+rowsets.emplace_back(new MockDiskRowSet(
StringPrintf("%010d", i * 2),
StringPrintf("%010d", i * 2 + 1)));
}
+ constexpr auto kBudgetMb = 128;
unordered_set picked;
- double quality = 0;
- NO_FATALS(RunTestCase(vec, /*size_budget_mb=*/128, , ));
- ASSERT_EQ(quality, 0);
+ double quality = 0.0;
+ NO_FATALS(RunTestCase(rowsets, kBudgetMb, , ));
+ ASSERT_EQ(quality, 0.0);
ASSERT_TRUE(picked.empty());
}
// Similar to the above, but with some small overlap between adjacent
// rowsets.
-TEST_F(TestCompactionPolicy, TestTinyOverlapRowsets) {
- if (!AllowSlowTests()) {
-LOG(INFO) << "Skipped";
-return;
- }
-
- RowSetVector vec;
- for (int i = 0; i < 1; i++) {
-vec.emplace_back(new MockDiskRowSet(
-StringPrintf("%010d", i * 1),
-StringPrintf("%010d", i * 1 + 11000)));
+TEST_F(TestCompactionPolicy, TestTinyOverlapRowSets) {
+ /* NB: Zero-padding of string keys omitted to save space.
+ *
+ * [0 -
[2/4] kudu git commit: Cleanup in rowset SVG layout dumping
Cleanup in rowset SVG layout dumping
I tested this by manually verifying that the /tablet-rowsetlayout-svg
endpoint and rowset layout SVG file dumping worked as expected.
Change-Id: I521fadf4336f9e11ea611b022cd4d8fcb269dfe2
Reviewed-on: http://gerrit.cloudera.org:8080/11612
Reviewed-by: Andrew Wong
Tested-by: Attila Bukor
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/77ead354
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/77ead354
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/77ead354
Branch: refs/heads/master
Commit: 77ead354b6ac86e3d251e35d9b0cf9da62979054
Parents: 891a8e3
Author: Will Berkeley
Authored: Sun Oct 7 21:56:14 2018 -0700
Committer: Will Berkeley
Committed: Tue Oct 9 15:10:20 2018 +
--
src/kudu/tablet/compaction_policy.cc | 2 +-
src/kudu/tablet/svg_dump.cc | 172 ++
src/kudu/tablet/svg_dump.h | 23 ++--
src/kudu/tablet/tablet.cc| 2 +-
4 files changed, 95 insertions(+), 104 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/77ead354/src/kudu/tablet/compaction_policy.cc
--
diff --git a/src/kudu/tablet/compaction_policy.cc
b/src/kudu/tablet/compaction_policy.cc
index 745f28b..d1ead15 100644
--- a/src/kudu/tablet/compaction_policy.cc
+++ b/src/kudu/tablet/compaction_policy.cc
@@ -471,7 +471,7 @@ Status BudgetedCompactionPolicy::PickRowSets(const
RowSetTree ,
}
picked->swap(best_solution.rowsets);
- DumpCompactionSVG(asc_min_key, *picked);
+ DumpCompactionSVGToFile(asc_min_key, *picked);
return Status::OK();
}
http://git-wip-us.apache.org/repos/asf/kudu/blob/77ead354/src/kudu/tablet/svg_dump.cc
--
diff --git a/src/kudu/tablet/svg_dump.cc b/src/kudu/tablet/svg_dump.cc
index 1427b9b..9c88655 100644
--- a/src/kudu/tablet/svg_dump.cc
+++ b/src/kudu/tablet/svg_dump.cc
@@ -17,8 +17,8 @@
#include "kudu/tablet/svg_dump.h"
+#include
#include
-#include
#include
#include
#include
@@ -26,13 +26,16 @@
#include
#include
-#include "kudu/gutil/gscoped_ptr.h"
#include "kudu/gutil/macros.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/stringprintf.h"
+#include "kudu/gutil/strings/substitute.h"
#include "kudu/gutil/strings/util.h"
#include "kudu/tablet/rowset_info.h"
+#include "kudu/util/env.h"
#include "kudu/util/flag_tags.h"
+#include "kudu/util/slice.h"
+#include "kudu/util/status.h"
// Flag to dump SVGs of every compaction decision.
//
@@ -40,7 +43,6 @@
// commands like:
// $ for x in compaction-*svg ; do convert $x $x.png ; done
// $ mencoder mf://compaction*png -mf fps=1 -ovc lavc -o compactions.avi
-
DEFINE_string(compaction_policy_dump_svgs_pattern, "",
"File path into which to dump SVG visualization of "
"selected compactions. This is mostly useful in "
@@ -49,10 +51,12 @@ DEFINE_string(compaction_policy_dump_svgs_pattern, "",
"with the compaction selection timestamp.");
TAG_FLAG(compaction_policy_dump_svgs_pattern, hidden);
+using std::endl;
using std::ostream;
using std::string;
using std::unordered_set;
using std::vector;
+using strings::Substitute;
namespace kudu {
namespace tablet {
@@ -63,35 +67,30 @@ namespace {
// distributes 'rowsets' into separate vectors in 'rows' such that
// within any given row, none of the rowsets overlap in keyspace.
void OrganizeSVGRows(const vector& candidates,
- vector >* rows) {
- rows->push_back(vector());
-
- for (const RowSetInfo : candidates) {
-// Slot into the first row of the output which fits it
+ vector>* rows) {
+ DCHECK(rows);
+ rows->clear();
+ for (const auto& candidate : candidates) {
+// Slot into the first row which fits it.
bool found_slot = false;
-for (vector : *rows) {
- // If this candidate doesn't intersect any other candidates in this
- // row, we can put it here.
- bool fits_in_row = true;
- for (const RowSetInfo *already_in_row : row) {
-if (candidate.Intersects(*already_in_row)) {
- fits_in_row = false;
- break;
-}
- }
+for (auto& row : *rows) {
+ // If this candidate doesn't intersect any other rowsets already in this
+ // row, we can put it in this row.
+ auto fits_in_row = std::none_of(row.begin(),
+ row.end(),
+ [](const RowSetInfo*
already_in_row) {
+return
candidate.Intersects(*already_in_row);
+ });
if (fits_in_row) {
row.push_back();
found_slot = true;
[1/4] kudu git commit: [docs] Add brief instructions on decommissioning tablet servers
Repository: kudu Updated Branches: refs/heads/master 89206ed91 -> 9ec8d28db [docs] Add brief instructions on decommissioning tablet servers Change-Id: I4e9ab976390ab6c0d5b8db0da00b27dc031037e5 Reviewed-on: http://gerrit.cloudera.org:8080/11618 Tested-by: Will Berkeley Reviewed-by: Andrew Wong Tested-by: Kudu Jenkins Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/891a8e3b Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/891a8e3b Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/891a8e3b Branch: refs/heads/master Commit: 891a8e3beca417007933bcae7c08067e4a3bdfb8 Parents: 89206ed Author: Will Berkeley Authored: Mon Oct 8 14:20:45 2018 -0700 Committer: Will Berkeley Committed: Tue Oct 9 14:54:01 2018 + -- docs/administration.adoc | 27 +++ 1 file changed, 27 insertions(+) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/891a8e3b/docs/administration.adoc -- diff --git a/docs/administration.adoc b/docs/administration.adoc index b176f58..1c3ab20 100644 --- a/docs/administration.adoc +++ b/docs/administration.adoc @@ -1249,3 +1249,30 @@ table, "RF" stands for "replication factor". If the rebalancer is running against a cluster where rebalancing replication factor one tables is not supported, it will rebalance all the other tables and the cluster as if those singly-replicated tables did not exist. + +[[tablet_server_decommissioning]] +.Decommissioning or Permanently Removing a Tablet Server From a Cluster + +Kudu does not currently have an automated way to remove a tablet server from +a cluster permanently. Instead, use the following steps: + +. Ensure the cluster is in good health using `ksck`. See <>. +. If the tablet server contains any replicas of tables with replication factor + 1, these replicas must be manually moved off the tablet server prior to + shutting it down. The `kudu tablet change_config move_replica` tool can be + used for this. +. Shut down the tablet server. After + `-follower_unavailable_considered_failed_sec`, which defaults to 5 minutes, + Kudu will begin to re-replicate the tablet server's replicas to other servers. + Wait until the process is finished. Progress can be monitored using `ksck`. +. Once all the copies are complete, `ksck` will continue to report the tablet + server as unavailable. The cluster will otherwise operate fine without the + tablet server. To completely remove it from the cluster so `ksck` shows the + cluster as completely healthy, restart the masters. In the case of a single + master, this will cause cluster downtime. With multimaster, restart the + masters in sequence to avoid cluster downtime. + +WARNING: Do not shut down multiple tablet servers at once. To remove multiple +tablet servers from the cluster, follow the above instructions for each tablet +server, ensuring that the previous tablet server is removed from the cluster and +`ksck` is healthy before shutting down the next.
