https://bz.apache.org/bugzilla/show_bug.cgi?id=60725
Bug ID: 60725
Summary: Allowing for slow-running test systems
Product: JMeter
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: minor
Priority: P2
Component: Main
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Some of the test cases make assumptions about how long a Thread.sleep() will
take.
However this can vary between systems due to timer granularity and how busy the
system is.
If the assumptions are wrong, then the test may fail unless the variation is
allowed for.
One way to do this is to allow a margin of error in the assertion.
However if a system runs very slowly then the error margin may have to be
increased so much for the test to pass that it no longer catches broken code.
This is/was the approach taken in many of the unit tests.
An alternative is to measure the actual sleep times, and calculate the expected
result accordingly.
This is what has been done in
http://svn.apache.org/viewvc?rev=1782600&view=rev
and
http://svn.apache.org/viewvc?rev=1782321&view=rev
Note that not all Thread.sleep() instances affect the outcome of tests.
Care must be taken to ensure that the test will still fail if the code is
broken, i.e. which sleeps are relevant to the test result.
Other sleeps may be necessary to ensure the test is working, but may not
directly affect the result.
In the case of TestSampleResult.testPauseTrue(), two of the sleeps represent
the sampler doing work, and another is the pause time where the sampler is not
supposed to record the time.
All 3 sleeps are necessary, but the pause time is not directly relevant as long
as it is long enough to ensure that the test would fail if the sampler
calculation were incorrect. e.g. if the pause time was too short (or zero) then
the test would not show that the sampler was correctly ignoring the time spent
in the paused state.
I fixed the two recent examples where recent CI tests failed, but there are
probably some more lurking.
--
You are receiving this mail because:
You are the assignee for the bug.
