The question: Are there particular performance cases that people would really like to see tracked for the OSAF hosted service?
The proposal: I'm currently planning on a short list of measured cases. Short as in probably 3 cases: PUT of a standard resource (1.5 sec), PROPFIND against a standard collection (0.2 sec), and ATOM GET of that same standard collection (4.0 sec). (Specific resources and collection TBD). Background ---------- This is a good deal different than what we see for Chandler performance targets, where we have 20 specific performance targets currently. This is because the goals of the targets are different. The Chandler and Cosmo performance targets are to measure different paths of code or different scales (10 events vs 1000 events), or to keep an eye on cases where optimization seems needed over time. The service targets are used essentially to figure out when the server is overloaded, and thus more or faster servers are needed. Performance of hosted services is kinda predictable; as servers gets loaded, pretty much everything slows down. You don't need to measure every little thing to know that "the server is slow". The logic behind the selection of these 3 targets (PUTs, PROPFIND, ATOM GET) are to exercise both read and write (as they can slow down at different rates on a server), and to test popular operations and subsystems. As operations change and we get real-world data, we can and should adjust these targets. It's possible we should include a EIM sync operation, for instance; I'm only excluding it today because I can't yet test it in production. The plan for these targets is that the tests will be scripted and run from the same colocation facility (to avoid open-Internet transients). The tests will be run on a regular schedule (probably every 5 minutes), and the total time for each test recorded and placed onto a graph/dashboard so that trends can be observed. (We'll see both intraday and long-term trends) for each test. Again, when the tests exceed their targets, that'll be a sign for us to seriously examine the server capacity and a plan of action would then be developed. Similar to Chandler performance targets, if we find an operation that is overly slow, we might add that to the list of regularly-measured tests. It should be noted again that the Hosted Service performance targets are and should be different than Cosmo performance targets. I don't think formal performance targets for Cosmo have been made, though we have identified and have bugs for some specific code paths (large-calendar deletion seems slow, for instance, as does the login sequence.) We know that the "login" use case for Cosmo is a major one right now. The login page itself and the initial calendar loading take longer than "satisfactory web page wait time" research would allow. There are a variety of enhancements being tracked for 0.6 which will definitely reduce this time, probably by an order of magnitude. While this is fodder for a Cosmo performance target, I'm not planning to track it as a service performance target. Currently, I'm planning to ignore the effect of caching. We'll be uploading the same resource or querying the same collection repeatedly, so cached results will differ from "fresh" transaction. But I think we can ignore this effect. We're more interested in trends than absolute numbers, and these operations will happen infrequently enough (5-10 minutes) as to probable not be in cache frequently (on a busy server at least). -- Jared Rhine <[EMAIL PROTECTED]> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Design" mailing list http://lists.osafoundation.org/mailman/listinfo/design
