Repository: kudu Updated Branches: refs/heads/master e3c5dd18c -> 5d4bf0c15
Fix flaky test TestRestartWithOrphanedReplicates Looks like the test was setting a fault injection flag before the setup which was causing the setup to fail. Moving the flag setting to after the setup, but before Start seems to have done the trick. We were getting about 1 failure per 1000 runs before this change, 0 per 1000 after. Dist test job after patch: http://dist-test.cloudera.org/job?job_id=efan.1495643022.19373 Dist test job before patch with failures: http://dist-test.cloudera.org/job?job_id=efan.1495149936.30555 Example from a failure log: I0518 23:26:00.447599 2346 master_service.cc:195] Signed X509 certificate for tserver {username='slave'} at 127.9.10.0:34667 W0518 23:26:00.448045 2529 fault_injection.cc:38] FAULT INJECTION ENABLED! W0518 23:26:00.448065 2529 fault_injection.cc:39] THIS SERVER MAY CRASH! E0518 23:26:00.448072 2529 fault_injection.cc:54] Injecting fault: FLAGS_fault_crash_before_append_commit (process will exit) I0518 23:26:00.448150 2510 heartbeater.cc:380] Master 127.0.0.1:38357 was elected leader, sending a full tablet report... W0518 23:26:00.450608 2331 connection.cc:462] server connection from 127.9.10.0:34667 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0518 23:26:00.450630 2319 connection.cc:462] client connection to 127.9.10.0:51332 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0518 23:26:00.450822 2331 connection.cc:462] client connection to 127.9.10.0:51332 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) F0518 23:26:30.315480 2314 test_workload.cc:266] Timed out: Timed out waiting for Table Creation *** Check failure stack trace: *** @ 0x7fb089e3b2fd google::LogMessage::Fail() at ??:0 @ 0x7fb089e3d1bd google::LogMessage::SendToLog() at ??:0 @ 0x7fb089e3ae39 google::LogMessage::Flush() at ??:0 @ 0x7fb089e3dc5f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fb0943b7492 kudu::TestWorkload::Setup() at ??:0 @ 0x40e68e kudu::TsRecoveryITest_TestRestartWithOrphanedReplicates_Test::TestBody() at /home/efan/src/kudu/build/release/../../src/kudu/integration-tests/ts_recovery-itest.cc:107 @ 0x7fb08ad35af8 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7fb08ad29fc2 testing::Test::Run() at ??:0 @ 0x7fb08ad2a108 testing::TestInfo::Run() at ??:0 @ 0x7fb08ad2a1e5 testing::TestCase::Run() at ??:0 @ 0x7fb08ad2a4c8 testing::internal::UnitTestImpl::RunAllTests() at ??:0 @ 0x7fb08ad36008 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7fb08ad2a7ad testing::UnitTest::Run() at ??:0 @ 0x7fb09416566c main at ??:0 @ 0x7fb088083f45 __libc_start_main at ??:0 @ 0x40dfb9 (unknown) at ??:? Change-Id: Ied9a55abd20841d350589ce56aa935ea1feece79 Reviewed-on: http://gerrit.cloudera.org:8080/6976 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin <[email protected]> Reviewed-by: Mike Percy <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/5d4bf0c1 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/5d4bf0c1 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/5d4bf0c1 Branch: refs/heads/master Commit: 5d4bf0c1586b60a23aebe47daa67d4f8bf8874f4 Parents: e3c5dd1 Author: Edward Fancher <[email protected]> Authored: Wed May 24 11:44:56 2017 -0500 Committer: Mike Percy <[email protected]> Committed: Thu May 25 20:23:15 2017 +0000 ---------------------------------------------------------------------- src/kudu/integration-tests/ts_recovery-itest.cc | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kudu/blob/5d4bf0c1/src/kudu/integration-tests/ts_recovery-itest.cc ---------------------------------------------------------------------- diff --git a/src/kudu/integration-tests/ts_recovery-itest.cc b/src/kudu/integration-tests/ts_recovery-itest.cc index 7ef8d49..8913140 100644 --- a/src/kudu/integration-tests/ts_recovery-itest.cc +++ b/src/kudu/integration-tests/ts_recovery-itest.cc @@ -82,8 +82,6 @@ void TsRecoveryITest::StartClusterOneTs(const vector<string>& extra_tserver_flag // inserted before the crash are recovered. TEST_F(TsRecoveryITest, TestRestartWithOrphanedReplicates) { NO_FATALS(StartClusterOneTs()); - cluster_->SetFlag(cluster_->tablet_server(0), - "fault_crash_before_append_commit", "0.05"); TestWorkload work(cluster_.get()); work.set_num_replicas(1); @@ -91,6 +89,10 @@ TEST_F(TsRecoveryITest, TestRestartWithOrphanedReplicates) { work.set_write_timeout_millis(100); work.set_timeout_allowed(true); work.Setup(); + + // Crash when the WAL contains a replicate message but no corresponding commit. + cluster_->SetFlag(cluster_->tablet_server(0), + "fault_crash_before_append_commit", "0.05"); work.Start(); // Wait for the process to crash due to the injected fault. @@ -101,6 +103,8 @@ TEST_F(TsRecoveryITest, TestRestartWithOrphanedReplicates) { // Restart the server, and it should recover. cluster_->tablet_server(0)->Shutdown(); + + // Restart the server and check to make sure that the change is eventually applied. ASSERT_OK(cluster_->tablet_server(0)->Restart());
