Thanks David. Taking state of the framework is an interesting design. I am assuming the scheduler is maintaining the state and then handing tasks on slaves. If that's the case, we can safely test executor (stateless - receiving event and returning appropriate status to the scheduler). You construct scheduler tests similarly by passing different states and event and observing the next state. This way you will be sure that your callbacks works fine in *isolation*. I would be concerned about the state of the framework in case of callback ordering (or re-execution) in *all possible scenarios*. Mocking is exactly what might uncover such bugs, but as you pointed out, I also think it would not be trivial for many frameworks.
At a high-level it would be important to know for frameworks developers that, 1. executors are working fine in isolation on fresh start, implementing the feature. 2. executors are working fine when rescheduled/restarted/in presence of other executors. 3. scheduler is working fine in isolation. 4. scheduler is fine in the wild ( in presence of others/failures/checkpointing/...). 1 is easy to do traditionally. 2 is possible if your executors do not have side effect or using Docker etc. 3 and 4 are not easy to do. I think having support/library for testing scheduler is something all the framework writer would benefit from. Not having to think about communication between executors and scheduler is already a big plus, can we also make it easier for developers to test about their scheduler behaviour? Thoughts? I would love to hear thoughts from others. Thanks, Dharmesh On Sun, Oct 12, 2014 at 8:03 PM, David Greenberg <dsg123456...@gmail.com> wrote: > For our frameworks, we don't tend to do much automated testing of the > Mesos interface--instead, we construct the framework state, then "send it a > message", since our callbacks take the state of the framework + the event > as the argument. This way, we don't need to have mesos running, and we can > trim away large amounts of code necessary to connect to mesos but > unnecessary for the actual feature under test. We've also been > experimenting with simulation testing by mocking out the mesos APIs. These > techniques are mostly effective when you can pretend that the executors > you're using don't communicate much, or when they're trivial to mock. > > On Sun, Oct 12, 2014 at 9:42 AM, Dharmesh Kakadia <dhkaka...@gmail.com> > wrote: > >> Hi, >> >> I am working on a tiny experimental framework for Mesos. I was wondering >> what is the recommended way of writing testcases for framework testing. I >> looked at the several existing frameworks, but its still not clear to me. I >> understand that I might be able to test executor functionality in isolation >> through normal test cases, but testing as a whole framework is what I am >> unclear about. >> >> Suggestions? Is that a non-goal? How do other framework developers go >> about it? >> >> Also, on the related note, is there a way to debug frameworks in better >> way than sifting through logs? >> >> Thanks, >> Dharmesh >> >> >> >
