Hi Dinesh, Thanks for the feedback. Please see below for my comments.
On Tue, Jul 19, 2016 at 11:53 PM, Dinesh J Weerakkody <[email protected]> wrote: > Hi Malith, > > > I'm just wondering whether this really matters or applicable for most of > the WSO2 products. What I believe is that the think time is important for > state-full applications but not for the state-less application such as most > of the WSO2 servers. > > > The think time is particularly important in stateful applications. However, in the case of stateless case we can consider this time as time between two requests/time between two successive events (= difference in time stamps). > What are we considering in our performance tests? Concurrency vs TPS, > Concurrency vs Latency. These numbers needs to be from the *server side*? > Isn’t it? > These numbers relates to server, however they are measured from outside (i.e. on the client side) > If it is the case what we need to do is load the server as much as > possible with the given no of users. So if we use a think time to test > cases we can have more concurrent users from the client side, but the > actual concurrent requests that we can process at a given time from the > server side will be the same. So the numbers we get might be wrong since we > use the user count from client side. > In the steady state: number of requests the server can process = arrival rate of requests into the server. The arrival rate will depend on concurrency (measured at the client side) and think time etc The behaviour that we normally notice is: As we increase the concurrency the throughput will increase up to a certain maximum value and stays steady (up to a certain concurrency level) and then starts to drop (means no longer in steady state). Once we include a think time in the performance test, we will expect to see similar behaviour. However, the concurrency level at which the maximum throughput is achieved will be higher and the system will be able to function in steady state for a larger range of concurrencies. One commonly used performance metric is the maximum number of concurrent users the system can support with the average latency or 90% latency percentile equal to or less than a given value. In such cases, if we know the think time (or have an idea of it), it make sense to include it in the performance test (if it is unknown we could perform tests by varying think time etc). This will give us an idea of the maximum number of *actual* concurrent users the system can support under this condition. So I think it make sense to consider think time in performance tests (at least in certain specific cases). However, we can do tests without it as well (as you have argued). > (Even now we have an unavoidable issue, network latency, which we need to > minimize as much as possible). > > > But, there are scenarios where we can use think time. The state-full > components such as APIM Store/Publisher components and WSO2 DS are the best > examples I can think of. Since we maintain sessions in these applications > it affect the memory. So when we add think time to these applications it > will allow many connections to the server and occupy more memory similar to > the real use cases. > > > WDYT? Please correct me if my understanding is wrong. > > > > Thanks > > *Dinesh J. Weerakkody* > Senior Software Engineer > WSO2 Inc. > lean | enterprise | middleware > M : +94 727 868676 | E : [email protected] | W : www.wso2.com > > On Mon, Jul 18, 2016 at 2:20 AM, Malith Jayasinghe <[email protected]> > wrote: > >> When evaluating the performance of products, it is important to perform >> these tests under realistic conditions. This is important since it allows >> us the understand the performance characteristics of the system in >> production environments. >> >> The think time plays an important role when doing performance tests. It >> is defined as the time between the completion of one request and the start >> of the next request. When generating requests (using load testing tools >> such as JMeter), we do not normally add a think time. This, however, may >> not represent the users’ real behaviour (access patterns) in the system. >> For example, when accessing a web site the users typically wait between >> page requests. >> >> >> Therefore, including a think time in the performance test makes the >> performance test more realistic as it represent users actual behaviour in >> the system (more closely). >> >> In most of the scenarios, the think time cannot be represented using a >> constant >> value and it is a random value which is distributed according a >> probability distribution function such as exponential distribution. >> >> The load on the system is determined by the number of requests being >> processed by the server. The number of active requests processed by the >> server will depend on both the number of concurrent users accessing the >> system and the think time. >> >> Under a given system load, an increase in the think time will result in >> the number of requests processed by the server to decrease and as a result >> the number of concurrent users the system can support will increase. >> >> We have done some performance tests on WSO2 APIM (under different number >> of concurrent users and different think times) and the analysis of these >> results can be found in [1]. >> >> >> [1] >> https://medium.com/@malith.jayasinghe/performance-testing-with-a-think-time-64b6b737e3f9#.aoxvs87er >> >> >> >> -- >> Malith Jayasinghe >> >> >> WSO2, Inc. (http://wso2.com) >> Email : [email protected] >> Mobile : 0770704040 >> Lean . Enterprise . Middleware >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Jayasinghe WSO2, Inc. (http://wso2.com) Email : [email protected] Mobile : 0770704040 Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
