[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user a-sure commented on the issue: https://github.com/apache/jmeter/pull/231 @vlsi, since thread departure schedule is pre-generated for defined timeframe, what about adding ability to log this schedule into file ? Another interesting feature would be to have real departure time logged, and\or warnings to to jmeter.log that there weren't any thread ready by the time in departure schedule - I mean possible case when SUT response time is way too slow and required request rate impossible to acheive with defined # of threads. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 @pmouawad , please review https://github.com/apache/jmeter/pull/369 ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hi @vsi, When do you think youâll be able to provide a documentation PR ? Thanks ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ham1 commented on the issue: https://github.com/apache/jmeter/pull/231 Is there a better name which makes the difference between the Constant Throughput Timer clearer, or should that be deprecated or just replaced with this? ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 "Precise Throughput Timer" sounds goods. PS. There's "Poisson Random Timer" in JMeter: http://jmeter.apache.org/usermanual/component_reference.html#Poisson_Random_Timer I guess it is used much less often. Frankly speaking, I do not see a use case for it. Should we somehow deprecate it? ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 @vlsi , What about: - Poisson Arrivals timer ? - Precise Throughput Timer ? ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hi @vsi, It looks ok to me, if you can add doc as separate PR it would be great ! Thanks ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 @pmouawad , thanks anyway for giving it a spin. Do you think it requires changes? If not, I think I can prepare a doc draft. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 > Philippe: It is just that as per your confirmation, one of the aims of this new component was: > Vladimir: "The very basic requirement is to ensure you end up exactly 50 iterations **per hour**. > Philippe: That's why I tested such things, so I think it needs to be mitigated a bit. Please, be careful: I suggested 50 per **hour**, and you are testing 50 per **second**. That is 3600 times higher rate than I suggested. I don't mean the timer works for 50/hour rate only, but 50/second is a bit different thing. >Now regarding "Repeatable test profile", for 2 runs, I don't see "similar test profile" Let us consider "10 second test with 1 per second on average". Current implementation writes "o.a.j.t.ConstantPoissonProcessGenerator: Generated ..." log record with expected timestamps of the samples. ``` 2017-12-23 21:39:01,449 INFO o.a.j.t.ConstantPoissonProcessGenerator: Generated 11 timings (ExponentialTimer 10 required, rate 1.0, duration 10, exact lim 2, i11) in 0 ms, restar t was issued 12 times. First 15 events will be fired at: 0.11401676510964737 (+0.11401676510964737), 0.6130748250774025 (+0.49905805996775515), 3.342639426533685 (+2.7295646014562824 ), 3.377172823174598 (+0.03453339664091315), 5.112423358789579 (+1.735250535614981), 5.1755972505795125 (+0.06317389178993338), 5.514713410628992 (+0.3391161600494792), 7.94457420086467 (+2.4298607902356784), 8.509235493336268 (+0.5646612924715981), 8.5888989163364 (+0.07966342300013096), 10.107204382018718 (+1.5183054656823192) ``` A bit more readable: 0.1, 0.6, 3.3, 3.4, 5.1, 5.2, 5.5, 7.9, 8.5, 8.6, 10.1 seconds. Note: there's exactly 10 samples in between 0..10 sec. Let me restart the test: ``` 2017-12-23 21:40:19,752 INFO o.a.j.t.ConstantPoissonProcessGenerator: Generated 11 timings (ExponentialTimer 10 required, rate 1.0, duration 10, exact lim 2, i11) in 0 ms, restart was issued 12 times. First 15 events will be fired at: 0.11401676510964737 (+0.11401676510964737), 0.6130748250774025 (+0.49905805996775515), 3.342639426533685 (+2.7295646014562824 ), 3.377172823174598 (+0.03453339664091315), 5.112423358789579 (+1.735250535614981), 5.1755972505795125 (+0.06317389178993338), 5.514713410628992 (+0.3391161600494792), 7.94457420086467 (+2.4298607902356784), 8.509235493336268 (+0.5646612924715981), 8.5888989163364 (+0.07966342300013096), 10.107204382018718 (+1.5183054656823192) ``` Note: the timestamps are exactly the same (the log timestamp is different, so it is a different execution). This is super-useful when testing low-rate things (e.g. 100 per hour) as it enables to have the same load profile, thus there's less moving parts when comparing different runs. Otherwise it is hard to tell if the difference in "CPU usage" profile is caused by different sample sequence or different system under test behavior. Note: in order for "repeatable test profile" to work, you need to set non-zero "randomSeed" (see https://github.com/apache/jmeter/pull/231/files#diff-76a5e8f118fb5e533f52ea80fd10dd0bR58 ) Unfortunately, JMeter is unable to initialize "test element property with a random-generated value". Test elements are created with constant values only, so I had to resort to a 0 being "random". I agree this is more like "easier for developer" rather than "easier for JMeter user", however I'm not sure what is JMeter way of doing that config. >Finally regarding Batched Departures, I am not sure to understand how it is supposed to influence load and if it is supposed to work correctly with First part in GUI. The idea is "batched departures" keep the load level (total number of samples per hour), however the samples are scheduled to come in pairs, triples, etc. For instance, with "batch size=2, throughput=10, duration=10" it produces samples as follows: 0.52, 0.52, 0.67, 0.67, 3.4, 3.4, 9.37, 9.37, 9.43, 9.43. Does the above make sense? PS. That is why I was saying [the documentation is missing](https://github.com/apache/jmeter/pull/231#issuecomment-342008122) ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hi @vlsi , Thanks for your clarifications. It is just that as per your confirmation, one of the aims of this new component was: **"The very basic requirement is to ensure you end up exactly 50 iterations per hour. Business customers would not understand if you report load test results with 47 executions "just because the random was random"."** That's why I tested such things, so I think it needs to be mitigated a bit. I tested with 1000 hits for 2 minutes and I get 1002 hits so ok. Now regarding "Repeatable test profile", for 2 runs, I don't see "similar test profile", but it is maybe just a matter of interpretation, what is exactly expected in your aim for "Repeatable test profile" Finally regarding Batched Departures, I am not sure to understand how it is supposed to influence load and if it is supposed to work correctly with First part in GUI. Thanks for clarifications. Regards ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >Is it really supposed to be exact ? Let me ask another question: "is thread scheduler=120sec supposed to make the test to last exactly 120 seconds"? Note: **timers do not limit the number of samples** in JMeter. If you want to limit the number of iterations (e.g. to make exactly 1 samples), you'd use additional counter or something like that. 1) When I try "startupdelay=0, duration=120sec, threads=10, throughput=1", I get 10011 samples in the aggregate report. I think it good enough to be considered exact. Note: target throughput is 1/120=83.3 samples/sec. 11samples/(1/120)=0.13 sec. That is those 11 "extra" samples can be due to 130ms spent on initialization/finalization of the test (e.g. in between "test started" and "threads started") 2) "exact number of samples" makes sense for cases like 10-100-1000 per HOUR, not 5000 per minute. I don't think there are requirements to aim for "exactly 1 per minute". In case there are that requirements, I would still discuss that in a separate PR as it would be tightly related with thread startup time, etc. Have you tried to test rates like 0.5...1 per second? ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hi @vlsi , Thanks for your feedback. I tried many configurations including thread delay to 0, and I never really reach exact sampling on duration. Is it really supposed to be exact ? https://user-images.githubusercontent.com/3127467/34321041-0999c088-e806-11e7-9409-657186ef92ba.png;> https://user-images.githubusercontent.com/3127467/34321042-09b21da4-e806-11e7-980f-0d8f0f9c15fc.png;> https://user-images.githubusercontent.com/3127467/34321043-09ca845c-e806-11e7-81c1-e92f53677bf5.png;> ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >Could you confirm your component implements the following features: That's true. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hi @vlsi , Could you confirm your component implements the following features: 1) "Specific number of iterations per hour". The very basic requirement is to ensure you end up exactly 50 iterations per hour. Business customers would not understand if you report load test results with 47 executions "just because the random was random". All timers except "Constant" can easily send excessive requests or send less requests than required. There is no way to control it. 2) "Bursty load" simulation. There is no easy way to test "50 iterations per hour as 10 bursts of 5 items". 3) "Repeatable test profile". All the random timers produce different pattern on each test run. This is not good for low-level analysis (e.g. compare of CPU% charts, etc). 4) "Avoid all thread groups" to fire at 00:00:00. By default, all thread groups would fire at 0, so there would be noticeable spike at the start of the test. Thanks ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >I will commit this PR unless there is a NO GO within 2 days. Philippe, thanks for looking into this. I'm afraid the documentation is missing. ---
Re: [GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Hi, No problem for me and very happy to have this PR merged in JMeter Antonio 2017-11-02 21:34 GMT+01:00 Philippe Mouawad: > Hi Antonio, > For the doc I can do it. > For the more intuitive GUI, can't we improve it later ? > Regards > > On Thu, Nov 2, 2017 at 9:23 PM, Antonio Gomes Rodrigues > wrote: > > > Hi, > > > > I think we need more doc and maybe a more intuitive GUI > > > > Antonio > > > > > > > > 2017-11-02 21:13 GMT+01:00 pmouawad : > > > > > Github user pmouawad commented on the issue: > > > > > > https://github.com/apache/jmeter/pull/231 > > > > > > Hello, > > > I will commit this PR unless there is a NO GO within 2 days. > > > > > > Thanks > > > > > > > > > --- > > > > > > > > > -- > Cordialement. > Philippe Mouawad. >
Re: [GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Hi Antonio, For the doc I can do it. For the more intuitive GUI, can't we improve it later ? Regards On Thu, Nov 2, 2017 at 9:23 PM, Antonio Gomes Rodrigueswrote: > Hi, > > I think we need more doc and maybe a more intuitive GUI > > Antonio > > > > 2017-11-02 21:13 GMT+01:00 pmouawad : > > > Github user pmouawad commented on the issue: > > > > https://github.com/apache/jmeter/pull/231 > > > > Hello, > > I will commit this PR unless there is a NO GO within 2 days. > > > > Thanks > > > > > > --- > > > -- Cordialement. Philippe Mouawad.
Re: [GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Hi, I think we need more doc and maybe a more intuitive GUI Antonio 2017-11-02 21:13 GMT+01:00 pmouawad: > Github user pmouawad commented on the issue: > > https://github.com/apache/jmeter/pull/231 > > Hello, > I will commit this PR unless there is a NO GO within 2 days. > > Thanks > > > --- >
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hello, I will commit this PR unless there is a NO GO within 2 days. Thanks ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >In my test plan I have 3 http request and one Exponential Timer at the same level That does not work like this. Please wrap "expenential timer" into some sampler (e.g. debug action or whatever). In JMeter all timers/listeners are inherited by sibling elements, thus when you place a timer at the same level as "3 http requests", you basically invoke the timer for each of the page, and that is something odd. You should have the plan as follows: ``` DebugSampler ExpTimer http1 http2 http3 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Hi, In my test plan I have 3 http request and one Exponential Timer at the same level When I run my script, only first http request are executed whereas with "Constant Throughput Timer", I don't have this problem ![poisson01](https://cloud.githubusercontent.com/assets/16882234/19597506/546a6a9e-9795-11e6-80ee-d037c2f67463.png) ![poisson02](https://cloud.githubusercontent.com/assets/16882234/19597541/81b8c388-9795-11e6-8950-85fc631f77a4.png) ![poisson03](https://cloud.githubusercontent.com/assets/16882234/19597546/85c9ee84-9795-11e6-80a7-4cb2e6ee169d.png) Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >I have some strange behavior in some test What is the behavior? Don't get me wrong, but I think the UI should be obvious. If the UI is obscure and/or the tooltip does not help, then it is a good point to update the UI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Yes I have read it but I have some strange behavior in some test It's why I would like to know how do you use it Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 @ra0077 , have you seen the tooltips for the input values of the timer? I'm sure those should explain all the things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Hi Vladimir, No idea for the UI Anybody have the answer? To help me to understand how to use it, can you provide me some JMeter jmx file for each goal ("Specific number of iterations per hour", "Bursty load") with a short description (in comment)? Thank Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 > Too much parameter Well, UI is built by JMeter OOB: https://github.com/apache/jmeter/pull/231/files#diff-fc396f633dbdaf0a98c5472613e9190dR25 Do you know if BeanInfoSupport can create "foldable" UI? >Do you plan to make some docs/examples ? docs -- sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Hi, My opinion is that the GUI is too complex because : Too much parameter You try to make too much different things in it And these things 1) "Specific number of iterations per hour". -> I think the GUI need to be like "Constant Throughput Timer", or better in Thread Group (and I think "Constant Throughput Timer" need to be in Thread Group too) 2) "Bursty load" I don't know which parameter to use 3) "Repeatable test profile" I don't know which parameter to use 4) "Avoid all thread groups" In my opinion, need to be in Thread Group In my opinion if you want to have all this functionality in only one element, a new GUI is necessary. e.g. A list to choose the goal ("Specific number of iterations per hour", "Bursty load", "Repeatable test profile","Avoid all thread groups" and Custom (like today)) When we select one, only needed parameter are displayed I have made some basic tests and I am impressed by the precision :-) Do you plan to make some docs/examples ? Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 > And if "test duration" parameter is not equal to expected test duration? Nothing breaks. The idea is as follows: each "test duration" the timer produces exact number of samples. For instance, if configured with "100 samples per hour, test duration 30 min", then 1) By 30min into the test there will be exactly 50 samples produced 2) By 60min since the beginning there will be exactly 100 samples produced 3) By 90min since the beginning there will be exactly 150 samples produced 4) ... However, the patterns between 0m..30m, 30m...60m, and 60m...90m will all be different. In other words, the timer will create different schedules for each 30m timeframe. > I think it's a GUI too complex and I search how to simplify it Yeah, the UI looks complicated, however I did try to remove non-required elements. On top of that, we might need to introduce "variable rate" somehow. For instance, in expression format: like `ramp(10/hour..100/hour in 10 min) keep(30min) ramp(..10/hour in 10 min)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Thank to the quick answer "so "test duration" parameter allows the timer to account for the expected test duration." And if "test duration" parameter is not equal to expected test duration? E.G. if I define a Loop count and not a Duration in the thread group I think it's a GUI too complex and I search how to simplify it I will do more test asap Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 >I don't understand how to use "Test duration" parameter The timer is random. Suppose one wants to have "100 samples per hour". Suppose the test duration is 30 minutes. The "expected" number of samples is 50. Do you remember that the timer is random. That basically means the number of produced samples for 30 minutes might be 50, it might be 40, and it might easily be 60. At the end of the day, poisson arrivals are not guaranteed to produce exactly 50 samples when running for 30 minutes under 100 samples/hour load. In practice, everybody wants "round" numbers in the load reports, so "test duartion" parameter allows the timer to account for the expected test duration. So, the timer ensures that the number of samples would be exactly 50. Does that make sense? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user ra0077 commented on the issue: https://github.com/apache/jmeter/pull/231 Hi, Thanks for the patch too. I don't understand how to use "Test duration" parameter Thank Antonio --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user vlsi commented on the issue: https://github.com/apache/jmeter/pull/231 > Any reason for not using ThreadLocalRandom.current() instead of Random ? TLR does not support setSeed. setSeed is required to get reproducible workloads. > https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadLocalRandom.html#setSeed(long) : Throws UnsupportedOperationException. Setting seeds in this generator is not supported. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] jmeter issue #231: WIP: timer that produces poisson arrivals with given cons...
Github user pmouawad commented on the issue: https://github.com/apache/jmeter/pull/231 Hello, Thanks for the patch. First review look good to me. Any reason for not using ThreadLocalRandom.current() instead of Random ? Regards Philippe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---