On 19 October 2013 02:17, Gil Tene <[email protected]> wrote: > >> [Trying again - please do not hijack this thread.] >> >> The Constant Throughput Timer (CTT) calculates the desired wait time, >> and if this is less than zero - i.e. a sample should already have been >> generated - it could trigger the creation of a failed Assertion (or similar) >> showing the time difference.
N.B. ^^^^^^^^^^^ >> >> Would this be sufficient to detect all CO occurrences? > > Two issues: > > 1. It would detect that CO probably happened, but not how much of it > happened, Missing 1msec or 1 minute will look the same. Huh? The time difference would be reported - is that not enough? > 2. It would only detect CO in test plans that include an actual CTT (Constant > Throughput Timer). It won't work for other timers, or when no timers are used Indeed, but in such cases is there any indication of the expected transaction rate? >> If not, what other metric needs to be checked? > > There are various things you can look for. > > Our OutlierCorrector work includes a pretty elaborate CO detector. It sits > either inline on the listener notification stream, or parses the log file. > The detector identifies sampler patterns, establishes expected interval > between patterns, and detects when the actual interval falls far above the > expected interval. This is a code change to JMeter, but a pretty localized > one. AIUI CO happens when the sample takes much longer than usual, thus (potentially) delaying any subseqent samples. Overlong samples can already be detected using a Delay Assertion. >> Even if it is not the only possible cause, would it be useful as a >> starting point? > > Yes. As a warning flag saying "throw away these test results". The results are not necessarily useless; at the very least they will show the load at which the system is starting to slow down. >> I am assuming that the CTT is the primary means of controlling the >> sample request rate. > > Unfortunately many test scenarios I've seen use other means. Many people use > other timers or other means for think time emulation. The CTT is not intended for think time emulation. It is intended to ensure a specific transaction rate. Think time obviously affects the transaction rate, but is not the sole determinant. >> If there are other elements that are commonly used to control the >> rate, please note them here. >> >> N.B: this thread is only for discussion of how to detect CO and how to >> report it. > > Reporting the existence of CO is an interesting starting point. But the only > right way to deal with such a report showing the existence of CO (with no > magnitude or other metrics) is to say " I guess the data I got is complete > crap, so all the stats and graphs I'm seeing mean nothing". I disagree that the output is useless. The delays are reported, so one can see how badly the test was affected. If the delays are all small, then a slight adjustment of the thread count and transaction rate should eliminate them. Only if the delays are large are the results less meaningful, though they can still show the base load at which the server starts to slow down. > If you can report "how much" CO you saw, As I wrote originally, the CTT would report the time diffence (i.e. delay). > it may help a bit in determining how bad the data is, and how the stats > should be treated by the reader. E.g. if you know that CO totaling some > amount of time X in a test of length Y had occured, then you know that any > percentile above (100 * (1-X)/Y) is completely bogus, and should be assumed > to be equal to the experienced max value. You can also take the approach that > the the rest of the percentiles should be shifted down by at least (100 * X > / Y). e.g. If you had CO that covered only 0.01% of the total test time, that > would be relatively good news. Exactly. > But if you had CO covering 5% of the test time, your measured 99%'ile is > actually the 94%'ile]. Averages are unfortunately anyone's guess when CO is > in play and not actually corrected for. > > Once you detect both the existence and the magnitude of CO, correcting for it > is actually pretty easy. The detection of "how much" is the semi-hard part. Detecting the delay using the CTT is trivial; it already has to calculate it to decide how long to wait. A negative wait is obviously a missed start time. Or am I missing something here? --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
