Repository: mesos Updated Branches: refs/heads/master 8e83b9a7c -> 1f231bf48
Documented how to expedite event firing. Review: https://reviews.apache.org/r/38160 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/1f231bf4 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/1f231bf4 Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/1f231bf4 Branch: refs/heads/master Commit: 1f231bf4807d7dfa74eb155f841cbaf50901b60c Parents: 8e83b9a Author: Alexander Rukletsov <[email protected]> Authored: Wed Oct 14 18:41:03 2015 +0200 Committer: Bernd Mathiske <[email protected]> Committed: Wed Oct 14 18:41:03 2015 +0200 ---------------------------------------------------------------------- docs/testing-patterns.md | 63 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/mesos/blob/1f231bf4/docs/testing-patterns.md ---------------------------------------------------------------------- diff --git a/docs/testing-patterns.md b/docs/testing-patterns.md index d0d92ff..e5fd989 100644 --- a/docs/testing-patterns.md +++ b/docs/testing-patterns.md @@ -6,6 +6,69 @@ layout: documentation A collection of common testing patterns used in Mesos tests. If you have found a good way to test a certain condition that you think may be useful for other cases, please document it here together with motivation and background. +## Expediting events with `Clock` +Some events in Mesos are separated by certain timeouts, for example framework registration attempts. Simple waiting for such events to fire leads to blocking the test thread for the duration of the associated timeout. This increases the duration of `make check` for no good reason. + +If an event is triggered by an act of processing a message from an actor's mailbox, it can be expedited with the help of libprocess' `Clock` routines. Delayed messages are maintained in sorted order by their due time and are dispatched â i.e. pushed into destination mailboxes â when this time comes. An important bit here is that time is driven by the internal libprocess clock. We can shift this clock into the future by calling `Clock::advance(<duration>)`, rendering certain front messages in the collection due now. These messages are dispatched instantly, effectively overriding the associated event's timeout. + +**NOTE**: Without calling `Clock::settle()` there is no guarantee a dispatched message has been already processed. + +Below is an example of this pattern. To avoid master backlogging, Mesos frameworks usually wait for some time (backoff) before retrying registration. In the test below we simulate the loss of a registration request, but avoid blocking the test for the backoff duration. + +~~~{.cpp} +TEST_F(FaultToleranceTest, FrameworkReliableRegistration) +{ + Try<PID<Master>> master = StartMaster(); + ASSERT_SOME(master); + + Try<PID<Slave>> slave = StartSlave(); + ASSERT_SOME(slave); + + // As a side effect of driver instantiation, registration backoff will be set + // to a default: internal::scheduler::REGISTRATION_BACKOFF_FACTOR. + MockScheduler sched; + MesosSchedulerDriver driver( + &sched, DEFAULT_FRAMEWORK_INFO, master.get(), DEFAULT_CREDENTIAL); + + Future<Nothing> registered; + EXPECT_CALL(sched, registered(&driver, _, _)) + .WillOnce(FutureSatisfy(®istered)); + + EXPECT_CALL(sched, resourceOffers(&driver, _)) + .WillRepeatedly(Return()); + + EXPECT_CALL(sched, offerRescinded(&driver, _)) + .Times(AtMost(1)); + + Future<AuthenticateMessage> authenticateMessage = + FUTURE_PROTOBUF(AuthenticateMessage(), _, master.get()); + + // Drop the first framework registered message, allow subsequent messages. + Future<FrameworkRegisteredMessage> frameworkRegisteredMessage = + DROP_PROTOBUF(FrameworkRegisteredMessage(), master.get(), _); + + driver.start(); + + // Ensure authentication occurs. + AWAIT_READY(authenticateMessage); + + AWAIT_READY(frameworkRegisteredMessage); + + // Trigger the registration retry instantly to avoid blocking the test. + Clock::pause(); + Clock::advance(internal::scheduler::REGISTRATION_BACKOFF_FACTOR); + + AWAIT_READY(registered); // Ensures registered message is received. + + driver.stop(); + driver.join(); + + Shutdown(); + + Clock::resume(); +} +~~~ + ## Using `Clock` magic to ensure an event is processed Scheduling a sequence of events in an asynchronous environment is not easy: a function call usually initiates an action and returns immediately, while the action runs in background. A simple, obvious, and bad solution is to use `os::sleep()` to wait for action completion. The time the action needs to finish may vary on different machines, while increasing sleep duration increases the test execution time and slows down `make check`. One of the right ways to do it is to wait for an action to finish and proceed right after. This is possible using libprocess' `Clock` routines.
