kudu git commit: [Java] Retry tests that don’t inherit from BaseKuduTest
Repository: kudu
Updated Branches:
refs/heads/master d6ee0047c -> 524485ebc
[Java] Retry tests that donât inherit from BaseKuduTest
The Scala tests and a couple special case tests
donât inherit from BaseKuduTest but may still
be flaky.
To fix this I moved the retry configuration logic into
the RetryRule and added the rule to the a few tests directly:
- TestSecurity
- TestKuduMetastorePlugin
Additionally I updated the Scala TestContext,
now called KuduTestSuite, to
use the JUnitSuite and include the RetryRule so
that all of the Scala tests would be retried. This
required a few syntax changes but has the added
benefit of not needing
`@RunWith(classOf[JUnitRunner])` at the top of
each test class. It also follows similar syntax to the
rest of our tests.
Change-Id: I596fde91a13f7e644fd1b1814e9a672f96fa0e4b
Reviewed-on: http://gerrit.cloudera.org:8080/11037
Reviewed-by: Adar Dembo
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/524485eb
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/524485eb
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/524485eb
Branch: refs/heads/master
Commit: 524485ebc797bf3e7e53d5b931cc0193e140ed41
Parents: d6ee004
Author: Grant Henke
Authored: Tue Jul 24 15:29:54 2018 -0500
Committer: Grant Henke
Committed: Wed Jul 25 00:22:21 2018 +
--
.../org/apache/kudu/backup/TestKuduBackup.scala | 16 +-
.../org/apache/kudu/client/BaseKuduTest.java| 10 +-
.../org/apache/kudu/client/TestSecurity.java| 7 +
.../java/org/apache/kudu/junit/RetryRule.java | 12 +-
java/kudu-hive/build.gradle | 1 +
java/kudu-hive/pom.xml | 7 +
.../hive/metastore/TestKuduMetastorePlugin.java | 5 +
.../kudu/spark/tools/ITBigLinkedListTest.scala | 12 +-
.../spark/tools/TestImportExportFiles.scala | 10 +-
.../kudu/spark/kudu/DefaultSourceTest.scala | 169 ++---
.../kudu/spark/kudu/KuduContextTest.scala | 17 +-
.../apache/kudu/spark/kudu/KuduRDDTest.scala| 10 +-
.../apache/kudu/spark/kudu/KuduTestSuite.scala | 181 +++
.../apache/kudu/spark/kudu/TestContext.scala| 172 --
14 files changed, 350 insertions(+), 279 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/524485eb/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
--
diff --git
a/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
b/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
index 1c166bd..b077b31 100644
---
a/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
+++
b/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
@@ -29,19 +29,20 @@ import org.apache.kudu.{ColumnSchema, Schema, Type}
import org.apache.kudu.spark.kudu._
import org.apache.kudu.util.DecimalUtil
import org.junit.Assert._
+import org.junit.Test
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner
-import org.scalatest.{FunSuite, Matchers}
import org.slf4j.{Logger, LoggerFactory}
import scala.collection.JavaConverters._
import scala.util.Random
@RunWith(classOf[JUnitRunner])
-class TestKuduBackup extends FunSuite with TestContext with Matchers {
+class TestKuduBackup extends KuduTestSuite {
val log: Logger = LoggerFactory.getLogger(getClass)
- test("Simple Backup and Restore") {
+ @Test
+ def testSimpleBackupAndRestore() {
insertRows(table, 100) // Insert data into the default test table.
backupAndRestore(tableName)
@@ -56,7 +57,8 @@ class TestKuduBackup extends FunSuite with TestContext with
Matchers {
assertTrue(partitionSchemasMatch(tA.getPartitionSchema,
tB.getPartitionSchema))
}
- test("Simple Backup and Restore Table Name With Special Characters") {
+ @Test
+ def testSimpleBackupAndRestoreWithSpecialCharacters() {
// Use an Impala-style table name to verify url encoding/decoding of the
table name works.
val impalaTableName = "impala::default.test"
@@ -73,7 +75,8 @@ class TestKuduBackup extends FunSuite with TestContext with
Matchers {
assert(rdd.isEmpty())
}
- test("Random Backup and Restore") {
+ @Test
+ def testRandomBackupAndRestore() {
Random.javaRandomToRandom(TestUtils.getRandom)
val table = createRandomTable()
@@ -93,7 +96,8 @@ class TestKuduBackup extends FunSuite with TestContext with
Matchers {
assertTrue(partitionSchemasMatch(tA.getPartitionSchema,
tB.getPartitionSchema))
}
- test("Backup and Restore Multiple Tables") {
+ @Test
+ def testBackupAndRestoreMultipleTables() {
val numRows = 1
val table1Name = "table1"
val table2Name =
kudu git commit: Add alter_external_catalogs flag to table rename tool
Repository: kudu
Updated Branches:
refs/heads/master 141ee5a52 -> d6ee0047c
Add alter_external_catalogs flag to table rename tool
It's going to be necessary to be able to alter the Kudu catalog
independently of the HMS in order to repair and upgrade tables when the
HMS integration is turned on. A follow-up commit will include a test
that exercises this flag when the HMS integration is enabled.
Change-Id: I9f5d97f234283b7740df727e085523c7f0b09735
Reviewed-on: http://gerrit.cloudera.org:8080/11017
Reviewed-by: Hao Hao
Tested-by: Dan Burkert
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d6ee0047
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d6ee0047
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d6ee0047
Branch: refs/heads/master
Commit: d6ee0047cef6fc5e3777045d50b1cfe861f745e9
Parents: 141ee5a
Author: Dan Burkert
Authored: Fri Jul 20 14:07:32 2018 -0700
Committer: Dan Burkert
Committed: Tue Jul 24 23:54:18 2018 +
--
src/kudu/client/client.h | 15 ---
src/kudu/tools/kudu-tool-test.cc | 14 +-
src/kudu/tools/tool_action_common.cc | 4
src/kudu/tools/tool_action_common.h | 4
src/kudu/tools/tool_action_hms.cc| 2 +-
src/kudu/tools/tool_action_table.cc | 5 +
6 files changed, 27 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/d6ee0047/src/kudu/client/client.h
--
diff --git a/src/kudu/client/client.h b/src/kudu/client/client.h
index 13e25da..cbf3257 100644
--- a/src/kudu/client/client.h
+++ b/src/kudu/client/client.h
@@ -58,19 +58,17 @@ class PartitionSchema;
class SecurityUnknownTskTest;
namespace client {
-class KuduClient;
+class KuduTableAlterer;
}
namespace tools {
class LeaderMasterProxy;
-
-Status AlterKuduTableOnly(client::KuduClient* kudu_client,
- const std::string& name,
- const std::string& new_name);
+void SetAlterExternalCatalogs(client::KuduTableAlterer*, bool);
} // namespace tools
namespace client {
+class KuduClient;
class KuduDelete;
class KuduInsert;
class KuduLoggingCallback;
@@ -79,7 +77,6 @@ class KuduScanBatch;
class KuduSession;
class KuduStatusCallback;
class KuduTable;
-class KuduTableAlterer;
class KuduTableCreator;
class KuduTablet;
class KuduTabletServer;
@@ -1193,11 +1190,7 @@ class KUDU_EXPORT KuduTableAlterer {
friend class KuduClient;
- friend Status tools::AlterKuduTableOnly(
- client::KuduClient* kudu_client,
- const std::string& name,
- const std::string& new_name);
-
+ friend void tools::SetAlterExternalCatalogs(KuduTableAlterer*, bool);
FRIEND_TEST(kudu::MasterHmsTest, TestAlterTable);
FRIEND_TEST(kudu::MasterHmsUpgradeTest, TestRenameExistingTables);
http://git-wip-us.apache.org/repos/asf/kudu/blob/d6ee0047/src/kudu/tools/kudu-tool-test.cc
--
diff --git a/src/kudu/tools/kudu-tool-test.cc b/src/kudu/tools/kudu-tool-test.cc
index d19dd15..f9ca66b 100644
--- a/src/kudu/tools/kudu-tool-test.cc
+++ b/src/kudu/tools/kudu-tool-test.cc
@@ -2013,6 +2013,10 @@ TEST_F(ToolTest, TestMasterList) {
TEST_F(ToolTest, TestRenameTable) {
NO_FATALS(StartExternalMiniCluster());
+ shared_ptr client;
+ ASSERT_OK(cluster_->CreateClient(nullptr, ));
+ string master_addr = cluster_->master()->bound_rpc_addr().ToString();
+
const string& kTableName = "kudu.table";
const string& kNewTableName = "kudu_table";
@@ -2022,17 +2026,17 @@ TEST_F(ToolTest, TestRenameTable) {
workload.set_num_replicas(1);
workload.Setup();
- string master_addr = cluster_->master()->bound_rpc_addr().ToString();
string out;
NO_FATALS(RunActionStdoutNone(Substitute("table rename_table $0 $1 $2",
master_addr, kTableName,
kNewTableName)));
- shared_ptr client;
- ASSERT_OK(KuduClientBuilder()
- .add_master_server_addr(master_addr)
- .Build());
shared_ptr table;
ASSERT_OK(client->OpenTable(kNewTableName, ));
+
+ NO_FATALS(RunActionStdoutNone(
+Substitute("table rename_table $0 $1 $2 --noalter_external_catalogs",
+ master_addr, kNewTableName, kTableName)));
+ ASSERT_OK(client->OpenTable(kTableName, ));
}
TEST_F(ToolTest, TestRenameColumn) {
http://git-wip-us.apache.org/repos/asf/kudu/blob/d6ee0047/src/kudu/tools/tool_action_common.cc
--
diff --git a/src/kudu/tools/tool_action_common.cc
b/src/kudu/tools/tool_action_common.cc
index e3bcc8f..c0e7301 100644
--- a/src/kudu/tools/tool_action_common.cc
+++
kudu git commit: Allow non-HMS compatible tables to be renamed after turning on HMS integration
Repository: kudu
Updated Branches:
refs/heads/master ccdb6b557 -> 7fe04438e
Allow non-HMS compatible tables to be renamed after turning on HMS integration
This is a bug fix followup to cee17c03bc30037bf2a, which introduced
automatic downcasing of table names during new table creation and table
renaming.
Change-Id: If554efbda06e33b5fdb40565e700bad8a306c143
Reviewed-on: http://gerrit.cloudera.org:8080/11016
Reviewed-by: Adar Dembo
Reviewed-by: Hao Hao
Tested-by: Dan Burkert
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7fe04438
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7fe04438
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7fe04438
Branch: refs/heads/master
Commit: 7fe04438ecd8e100509ba224826e753ffc851d49
Parents: ccdb6b5
Author: Dan Burkert
Authored: Tue Jul 17 17:33:00 2018 -0700
Committer: Dan Burkert
Committed: Tue Jul 24 21:14:07 2018 +
--
src/kudu/client/client.h | 2 ++
src/kudu/integration-tests/master_hms-itest.cc | 29 +++
src/kudu/master/catalog_manager.cc | 31 -
3 files changed, 49 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/7fe04438/src/kudu/client/client.h
--
diff --git a/src/kudu/client/client.h b/src/kudu/client/client.h
index 9ad7afc..13e25da 100644
--- a/src/kudu/client/client.h
+++ b/src/kudu/client/client.h
@@ -52,6 +52,7 @@ namespace kudu {
class ClientStressTest_TestUniqueClientIds_Test;
class KuduPartialRow;
class MasterHmsTest_TestAlterTable_Test;
+class MasterHmsUpgradeTest_TestRenameExistingTables_Test;
class MonoDelta;
class PartitionSchema;
class SecurityUnknownTskTest;
@@ -1198,6 +1199,7 @@ class KUDU_EXPORT KuduTableAlterer {
const std::string& new_name);
FRIEND_TEST(kudu::MasterHmsTest, TestAlterTable);
+ FRIEND_TEST(kudu::MasterHmsUpgradeTest, TestRenameExistingTables);
KuduTableAlterer(KuduClient* client,
const std::string& name);
http://git-wip-us.apache.org/repos/asf/kudu/blob/7fe04438/src/kudu/integration-tests/master_hms-itest.cc
--
diff --git a/src/kudu/integration-tests/master_hms-itest.cc
b/src/kudu/integration-tests/master_hms-itest.cc
index 4706a9d..7ba3c8c 100644
--- a/src/kudu/integration-tests/master_hms-itest.cc
+++ b/src/kudu/integration-tests/master_hms-itest.cc
@@ -579,4 +579,33 @@ TEST_F(MasterHmsUpgradeTest,
TestConflictingNormalizedNames) {
std::sort(tables.begin(), tables.end());
ASSERT_EQ(tables, vector({ "default.MyTable",
"default.mytable-renamed" }));
}
+
+// Checks that existing tables with HMS-incompatible names can be renamed post
+// upgrade using a Kudu-catalog only alter.
+TEST_F(MasterHmsUpgradeTest, TestRenameExistingTables) {
+ ASSERT_OK(CreateKuduTable("default", "UPPERCASE"));
+ ASSERT_OK(CreateKuduTable("default", "illegal-charsâ"));
+
+ // Shutdown the masters and turn on the HMS integration
+ cluster_->ShutdownNodes(cluster::ClusterNodes::MASTERS_ONLY);
+ cluster_->EnableMetastoreIntegration();
+ ASSERT_OK(cluster_->Restart());
+
+ vector tables;
+ ASSERT_OK(client_->ListTables());
+ std::sort(tables.begin(), tables.end());
+ ASSERT_EQ(tables, vector({ "default.UPPERCASE",
"default.illegal-charsâ" }));
+
+ // Rename the tables using a Kudu catalog only rename.
+ unique_ptr
alterer(client_->NewTableAlterer("default.UPPERCASE"));
+
ASSERT_OK(alterer->RenameTo("default.uppercase")->alter_external_catalogs(false)->Alter());
+
+ alterer.reset(client_->NewTableAlterer("default.illegal-charsâ"));
+
ASSERT_OK(alterer->RenameTo("default.illegal_chars")->alter_external_catalogs(false)->Alter());
+
+ tables.clear();
+ client_->ListTables();
+ std::sort(tables.begin(), tables.end());
+ ASSERT_EQ(tables, vector({ "default.illegal_chars",
"default.uppercase" }));
+}
} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/7fe04438/src/kudu/master/catalog_manager.cc
--
diff --git a/src/kudu/master/catalog_manager.cc
b/src/kudu/master/catalog_manager.cc
index b10a4cc..d6b4654 100644
--- a/src/kudu/master/catalog_manager.cc
+++ b/src/kudu/master/catalog_manager.cc
@@ -2261,7 +2261,7 @@ Status CatalogManager::AlterTable(const
AlterTableRequestPB& req,
resp, MasterErrorPB::TABLE_NOT_FOUND);
}
- string table_name = l.data().name();
+ string normalized_table_name = NormalizeTableName(l.data().name());
*resp->mutable_table_id() = table->id();
// 3. Calculate and validate new schema for the on-disk state, not persisted
yet.
@@ -2288,22 +2288,27 @@ Status
kudu git commit: KUDU-2510 Fix symmetric difference logging
Repository: kudu
Updated Branches:
refs/heads/master 5b09a693d -> 7aab411d3
KUDU-2510 Fix symmetric difference logging
When the on-disk and provided master_addresses don't match, the error
message was misleading as it showed them swapped, i.e. the
--master_addresses lists 3 masters, and there's 1 in the Raft config, it
showed "on-disk master list (master1:7051, master2:7051, master3:7051)
and provided master list (master1:7051) differ."
This commit swaps these two lists.
Change-Id: I1fd5a0aaa5bd1398d874a8526240b12c37baad70
Reviewed-on: http://gerrit.cloudera.org:8080/11031
Reviewed-by: Will Berkeley
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7aab411d
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7aab411d
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7aab411d
Branch: refs/heads/master
Commit: 7aab411d3187ea066c935af0215f894c5eca6aae
Parents: 5b09a69
Author: Attila Bukor
Authored: Tue Jul 24 16:11:26 2018 +0200
Committer: Attila Bukor
Committed: Tue Jul 24 15:39:17 2018 +
--
src/kudu/master/sys_catalog.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/7aab411d/src/kudu/master/sys_catalog.cc
--
diff --git a/src/kudu/master/sys_catalog.cc b/src/kudu/master/sys_catalog.cc
index dd36fae..01b18f1 100644
--- a/src/kudu/master/sys_catalog.cc
+++ b/src/kudu/master/sys_catalog.cc
@@ -221,8 +221,8 @@ Status SysCatalogTable::Load(FsManager *fs_manager) {
string msg = Substitute(
"on-disk master list ($0) and provided master list ($1) differ. "
"Their symmetric difference is: $2",
- JoinStrings(peer_addrs_from_opts, ", "),
JoinStrings(peer_addrs_from_disk, ", "),
+ JoinStrings(peer_addrs_from_opts, ", "),
JoinStrings(symm_diff, ", "));
return Status::InvalidArgument(msg);
}
