kudu git commit: KUDU-1882 Configuration improvements for Flume Kudu Sink regexp operations producer.
Repository: kudu
Updated Branches:
refs/heads/master 4fba6e8a1 -> 6030c5a32
KUDU-1882 Configuration improvements for Flume Kudu Sink regexp operations
producer.
Adding new properties for the different parsing error policies.
Deprecating the old ones. Default value constants are not removed as they are
public variables on a public class and it would be an API change.
Adding new test class to test the configuration and behaviour.
Change-Id: I0b352b27583ae9eee6cdbe28a768362bea36e00f
Reviewed-on: http://gerrit.cloudera.org:8080/11391
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/6030c5a3
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/6030c5a3
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/6030c5a3
Branch: refs/heads/master
Commit: 6030c5a32dd0e0c825e8c4f9152a66253de5474c
Parents: 4fba6e8
Author: Ferenc Szabó
Authored: Wed Sep 5 19:21:17 2018 +0200
Committer: Mike Percy
Committed: Tue Sep 25 21:35:06 2018 +
--
.../sink/RegexpKuduOperationsProducer.java | 141 --
...expKuduOperationsProducerParseErrorTest.java | 273 +++
2 files changed, 393 insertions(+), 21 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/6030c5a3/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducer.java
--
diff --git
a/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducer.java
b/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducer.java
index ff9fd3c..a3b2c57 100644
---
a/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducer.java
+++
b/java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducer.java
@@ -21,6 +21,7 @@ package org.apache.kudu.flume.sink;
import java.math.BigDecimal;
import java.nio.charset.Charset;
+import java.util.Arrays;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -100,7 +101,9 @@ import org.apache.kudu.client.Upsert;
* producer.skipMissingColumn
* false
* No
- * What to do if a column in the Kudu table has no corresponding capture
group.
+ *
+ * @deprecated use {@code producer.missingColumnPolicy}
+ * What to do if a column in the Kudu table has no corresponding capture
group.
* If set to {@code true}, a warning message is logged and the operation is
still attempted.
* If set to {@code false}, an exception is thrown and the sink will not
process the
* {@code Event}, causing a Flume {@code Channel} rollback.
@@ -109,7 +112,9 @@ import org.apache.kudu.client.Upsert;
* producer.skipBadColumnValue
* false
* No
- * What to do if a value in the pattern match cannot be coerced to the
required type.
+ *
+ * @deprecated use {@code producer.badColumnValuePolicy}
+ * What to do if a value in the pattern match cannot be coerced to the
required type.
* If set to {@code true}, a warning message is logged and the operation is
still attempted.
* If set to {@code false}, an exception is thrown and the sink will not
process the
* {@code Event}, causing a Flume {@code Channel} rollback.
@@ -118,9 +123,44 @@ import org.apache.kudu.client.Upsert;
* producer.warnUnmatchedRows
* true
* No
- * Whether to log a warning about payloads that do not match the
pattern. If set to
+ *
+ * @deprecated use {@code producer.unmatchedRowPolicy}
+ * Whether to log a warning about payloads that do not match the pattern. If
set to
* {@code false}, event bodies with no matches will be silently dropped.
*
+ *
+ * producer.missingColumnPolicy
+ * REJECT
+ * No
+ * What to do if a column in the Kudu table has no corresponding capture
group.
+ * If set to {@code REJECT}, an exception is thrown and the sink will not
process the
+ * {@code Event}, causing a Flume {@code Channel} rollback.
+ * If set to {@code WARN}, a warning message is logged and the operation is
still produced.
+ * If set to {@code IGNORE}, the operation is still produced without any log
message.
+ *
+ *
+ * producer.badColumnValuePolicy
+ * REJECT
+ * No
+ * What to do if a value in the pattern match cannot be coerced to the
required type.
+ * If set to {@code REJECT}, an exception is thrown and the sink will not
process the
+ * {@code Event}, causing a Flume {@code Channel} rollback.
+ * If set to {@code WARN}, a warning message is logged and the operation is
still produced,
+ * but does not include the given column.
+ * If set to {@code IGNORE}, the operation is still produced, but does not
include the given
+ * column
kudu git commit: java: improve assertion failure message in TestAsyncKuduSession.test
Repository: kudu
Updated Branches:
refs/heads/master 6030c5a32 -> 9130bb0e1
java: improve assertion failure message in TestAsyncKuduSession.test
This test is fairly flaky (tracked in KUDU-1521) and while I don't have
time to fix the underlying issue I want to make it easier to track this
in the future. Instead of just throwing a base AssertionError, it will
now throw an error indicating that it failed because it was expecting a
PleaseThrottleException that never appeared.
Change-Id: If8b2f300f34ec70ee48d81a96f378587de9f8a04
Reviewed-on: http://gerrit.cloudera.org:8080/11510
Reviewed-by: Adar Dembo
Tested-by: Mike Percy
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/9130bb0e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/9130bb0e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/9130bb0e
Branch: refs/heads/master
Commit: 9130bb0e15d25ea8e1dfde28bebc752793100618
Parents: 6030c5a
Author: Mike Percy
Authored: Tue Sep 25 12:46:53 2018 -0700
Committer: Mike Percy
Committed: Tue Sep 25 21:42:10 2018 +
--
.../src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/9130bb0e/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java
--
diff --git
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java
index 0e99254..1bf8c63 100644
---
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java
+++
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java
@@ -449,7 +449,7 @@ public class TestAsyncKuduSession extends BaseKuduTest {
}
session.flush().join(DEFAULT_SLEEP);
assertEquals(20, countInRange(151, 171));
-assertTrue(gotException);
+assertTrue("Expected PleaseThrottleException, but it was never thrown",
gotException);
}
private Insert createInsert(int key) {
[2/2] kudu git commit: Add Hive Metastore service principal configuration
Add Hive Metastore service principal configuration
This commit adds a new flag, --hive_metastore_kerberos_principal flag
which corresponds to the HMS 'hive.metastore.kerberos.principal'
configuration. This configuration is rarely overridden, but in cases
where it is, having a way to match it in Kudu is critical.
Change-Id: I6c5c56423a62cba0b696f97d866077b35845ce26
Reviewed-on: http://gerrit.cloudera.org:8080/11503
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/a09c89ab
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/a09c89ab
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/a09c89ab
Branch: refs/heads/master
Commit: a09c89abc1c0df62cffc9bdf3b36eede9150c4b9
Parents: 24b38f7
Author: Dan Burkert
Authored: Mon Sep 24 16:41:27 2018 -0700
Committer: Dan Burkert
Committed: Tue Sep 25 17:42:49 2018 +
--
src/kudu/hms/hms_catalog-test.cc| 1 +
src/kudu/hms/hms_catalog.cc | 7 +++
src/kudu/hms/hms_client-test.cc | 5 -
src/kudu/integration-tests/master_hms-itest.cc | 1 +
src/kudu/mini-cluster/external_mini_cluster-test.cc | 1 +
src/kudu/thrift/client.cc | 4 +++-
src/kudu/thrift/client.h| 6 ++
src/kudu/thrift/sasl_client_transport.cc| 9 +
src/kudu/thrift/sasl_client_transport.h | 6 +-
src/kudu/tools/kudu-tool-test.cc| 3 +++
10 files changed, 36 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/a09c89ab/src/kudu/hms/hms_catalog-test.cc
--
diff --git a/src/kudu/hms/hms_catalog-test.cc b/src/kudu/hms/hms_catalog-test.cc
index a2bfabf..83369fb 100644
--- a/src/kudu/hms/hms_catalog-test.cc
+++ b/src/kudu/hms/hms_catalog-test.cc
@@ -184,6 +184,7 @@ class HmsCatalogTest : public KuduTest {
ASSERT_OK(kdc_->Kinit("alice"));
ASSERT_OK(kdc_->SetKrb5Environment());
hms_client_opts.enable_kerberos = true;
+ hms_client_opts.service_principal = "hive";
// Configure the HmsCatalog flags.
FLAGS_hive_metastore_sasl_enabled = true;
http://git-wip-us.apache.org/repos/asf/kudu/blob/a09c89ab/src/kudu/hms/hms_catalog.cc
--
diff --git a/src/kudu/hms/hms_catalog.cc b/src/kudu/hms/hms_catalog.cc
index aeaf226..ae4d998 100644
--- a/src/kudu/hms/hms_catalog.cc
+++ b/src/kudu/hms/hms_catalog.cc
@@ -73,6 +73,12 @@ DEFINE_bool(hive_metastore_sasl_enabled, false,
"When enabled, the --keytab_file flag must be provided.");
TAG_FLAG(hive_metastore_sasl_enabled, experimental);
+DEFINE_string(hive_metastore_kerberos_principal, "hive",
+ "The service principal of the Hive Metastore server. Must match "
+ "the primary (user) portion of hive.metastore.kerberos.principal
option "
+ "in the Hive Metastore configuration.");
+TAG_FLAG(hive_metastore_kerberos_principal, experimental);
+
DEFINE_int32(hive_metastore_retry_count, 1,
"The number of times that HMS operations will retry after "
"encountering retriable failures, such as network errors.");
@@ -416,6 +422,7 @@ Status HmsCatalog::Reconnect() {
options.recv_timeout =
MonoDelta::FromSeconds(FLAGS_hive_metastore_recv_timeout);
options.conn_timeout =
MonoDelta::FromSeconds(FLAGS_hive_metastore_conn_timeout);
options.enable_kerberos = FLAGS_hive_metastore_sasl_enabled;
+ options.service_principal = FLAGS_hive_metastore_kerberos_principal;
options.max_buf_size = FLAGS_hive_metastore_max_message_size;
// Try reconnecting to each HMS in sequence, returning the first one which
http://git-wip-us.apache.org/repos/asf/kudu/blob/a09c89ab/src/kudu/hms/hms_client-test.cc
--
diff --git a/src/kudu/hms/hms_client-test.cc b/src/kudu/hms/hms_client-test.cc
index e677cd0..9091a46 100644
--- a/src/kudu/hms/hms_client-test.cc
+++ b/src/kudu/hms/hms_client-test.cc
@@ -127,6 +127,7 @@ TEST_P(HmsClientTest, TestHmsOperations) {
ASSERT_OK(kdc.Kinit("alice"));
ASSERT_OK(kdc.SetKrb5Environment());
hms_client_opts.enable_kerberos = true;
+hms_client_opts.service_principal = "hive";
}
ASSERT_OK(hms.Start());
@@ -288,7 +289,8 @@ TEST_P(HmsClientTest, TestLargeObjects) {
if (protection) {
ASSERT_OK(kdc.Start());
-string spn = "hive/127.0.0.1";
+// Try a non-standard service principal to ensure it works correctly.
+string spn = "hive_alternate_sp/127.0.0.1";
string ktpath;
ASSERT_OK(kdc.CreateServiceKeytab(spn, ));
@@ -302,6
[1/2] kudu git commit: hybrid_clock: restore SleepFor in WaitUntilAfterLocally
Repository: kudu
Updated Branches:
refs/heads/master ddaa766e2 -> a09c89abc
hybrid_clock: restore SleepFor in WaitUntilAfterLocally
This seems to have gone missing from commit f2d96437c. The end result is a
busy loop, which was probably unintentional. I also removed the loop, which
seemed unnecessary with the SleepFor.
To test, I wrote a small program that scanned a table at Now() + 10s, after
setting --scanner_max_wait_ms to 1 (default is 1000). Although the wall
clock time was the same, the cycle count was far higher per perf stat.
Change-Id: Iecbb077b1f88293dcb0ab53e40ff3862f772694e
Reviewed-on: http://gerrit.cloudera.org:8080/11495
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/24b38f73
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/24b38f73
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/24b38f73
Branch: refs/heads/master
Commit: 24b38f73739ef76f69fbe950ff103dc6c1b3e6a8
Parents: ddaa766
Author: Adar Dembo
Authored: Fri Sep 21 12:27:41 2018 -0700
Committer: David Ribeiro Alves
Committed: Mon Sep 24 15:03:30 2018 +
--
src/kudu/clock/hybrid_clock.cc | 32
1 file changed, 16 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/24b38f73/src/kudu/clock/hybrid_clock.cc
--
diff --git a/src/kudu/clock/hybrid_clock.cc b/src/kudu/clock/hybrid_clock.cc
index b97fd2d..7e87d4d 100644
--- a/src/kudu/clock/hybrid_clock.cc
+++ b/src/kudu/clock/hybrid_clock.cc
@@ -291,31 +291,31 @@ Status HybridClock::WaitUntilAfter(const Timestamp& then,
SleepFor(MonoDelta::FromMicroseconds(wait_for_usec));
-
VLOG(1) << "WaitUntilAfter(): Incoming time(latest): " << then_latest_usec
<< " Now(earliest): " << now_earliest_usec << " error: " << error
<< " Waiting for: " << wait_for_usec;
-
return Status::OK();
}
Status HybridClock::WaitUntilAfterLocally(const Timestamp& then,
const MonoTime& deadline) {
- while (true) {
-Timestamp now;
-uint64_t error;
-{
- std::lock_guard lock(lock_);
- NowWithError(, );
-}
-if (now > then) {
- return Status::OK();
-}
-uint64_t wait_for_usec = GetPhysicalValueMicros(then) -
GetPhysicalValueMicros(now);
-
-// Check that sleeping wouldn't sleep longer than our deadline.
-RETURN_NOT_OK(CheckDeadlineNotWithinMicros(deadline, wait_for_usec));
+ Timestamp now;
+ uint64_t error;
+ {
+std::lock_guard lock(lock_);
+NowWithError(, );
}
+ if (now > then) {
+return Status::OK();
+ }
+ uint64_t wait_for_usec = GetPhysicalValueMicros(then) -
GetPhysicalValueMicros(now);
+
+ // Check that sleeping wouldn't sleep longer than our deadline.
+ RETURN_NOT_OK(CheckDeadlineNotWithinMicros(deadline, wait_for_usec));
+
+ SleepFor(MonoDelta::FromMicroseconds(wait_for_usec));
+
+ return Status::OK();
}
bool HybridClock::IsAfter(Timestamp t) {
kudu git commit: [tablet_server-test] cleaner exit-on-failure in TestStatus
Repository: kudu
Updated Branches:
refs/heads/master a09c89abc -> 4fba6e8a1
[tablet_server-test] cleaner exit-on-failure in TestStatus
Updated the TestStatus scenario of the TabletServerTest to terminate
cleaner and faster in case of a failure when restarting tablet server.
Prior to this fix, when the mini tablet server fails to restart,
the test scenario would run indefinitely or until some higher-level
test framework terminates the tablet_server-test process.
As for the reason behind the failure on restart, the mini tablet
server might fail to restart if one the ports it tries to bind to
is used by another process.
Change-Id: I7c5efef9e579f6fd357357517c32a26440896967
Reviewed-on: http://gerrit.cloudera.org:8080/11506
Tested-by: Alexey Serbin
Reviewed-by: Adar Dembo
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/4fba6e8a
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/4fba6e8a
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/4fba6e8a
Branch: refs/heads/master
Commit: 4fba6e8a16378ffc00bd399ca65161e96f9ce20e
Parents: a09c89a
Author: Alexey Serbin
Authored: Mon Sep 24 19:25:48 2018 -0700
Committer: Alexey Serbin
Committed: Tue Sep 25 18:14:03 2018 +
--
src/kudu/tserver/tablet_server-test.cc | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/4fba6e8a/src/kudu/tserver/tablet_server-test.cc
--
diff --git a/src/kudu/tserver/tablet_server-test.cc
b/src/kudu/tserver/tablet_server-test.cc
index 084da03..5217fe0 100644
--- a/src/kudu/tserver/tablet_server-test.cc
+++ b/src/kudu/tserver/tablet_server-test.cc
@@ -234,6 +234,7 @@ TEST_F(TabletServerTest, TestStatus) {
}
});
SCOPED_CLEANUP({
+latch.CountDown();
status_thread.join();
});
@@ -244,9 +245,6 @@ TEST_F(TabletServerTest, TestStatus) {
mini_server_->Shutdown();
ASSERT_OK(mini_server_->Restart());
}
-
- // All done, stop the test thread.
- latch.CountDown();
}
TEST_F(TabletServerTest, TestServerClock) {
