On 3/5/21 4:03 AM, Vladimir Sementsov-Ogievskiy wrote:
05.03.2021 04:22, John Snow wrote:
On 3/4/21 5:17 AM, Vladimir Sementsov-Ogievskiy wrote:
Sometimes one of cells in a testing table runs too slow. And we really
don't want to wait so long. Limit number of runs in this case.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsement...@virtuozzo.com>
---
scripts/simplebench/simplebench.py | 29 +++++++++++++++++++++++++----
1 file changed, 25 insertions(+), 4 deletions(-)
diff --git a/scripts/simplebench/simplebench.py
b/scripts/simplebench/simplebench.py
index f61513af90..b153cae274 100644
--- a/scripts/simplebench/simplebench.py
+++ b/scripts/simplebench/simplebench.py
@@ -19,9 +19,11 @@
#
import statistics
+import time
-def bench_one(test_func, test_env, test_case, count=5,
initial_run=True):
+def bench_one(test_func, test_env, test_case, count=5,
initial_run=True,
+ slow_limit=100):
"""Benchmark one test-case
test_func -- benchmarking function with prototype
@@ -36,6 +38,8 @@ def bench_one(test_func, test_env, test_case,
count=5, initial_run=True):
test_case -- test case - opaque second argument for test_func
count -- how many times to call test_func, to calculate
average
initial_run -- do initial run of test_func, which don't get
into result
+ slow_limit -- reduce test runs to 2, if current run exceedes
the limit
+ (in seconds)
s/exceedes/exceeds, and you need to mention that if the initial run
exceeds the limit, it will change the behavior to count that result.
It is also possible (conceivably) that the initial run exceeds the
limit, but subsequent runs don't, so it might be hard to predict how
many tests it'll actually run.
If you're OK with that behavior, maybe:
"Consider a test run 'slow' once it exceeds this limit, in seconds.
Stop early once there are two 'slow' runs, including the initial run.
Slow initial runs will be included in the results."
Lastly, this will change existing behavior -- do we care? Should it
default to None instead? Should we be able to pass None or 0 to
disable this behavior?
For sure I don't care about changing the behavior. Consider simplebench
in a version 0.0.1 :). Maybe, I should make a comment somewhere, but
nobody will read it anyway.
Yep, it's yours anyway. Just thought I'd mention it. It's probably the
case that you're the only person who actually uses this at the moment.
The aim of the patch is to minimize waiting for too long cells of the
table, which are obviously too much longer then the others. Probably the
logic should be improved a bit about ignoring or using initial-run result..
Like this:
If both initial and first run are slow, count both and stop here.
Otherwise, stop at first slow normal run and don't count initial run.
Or may be even
If both initial and first run are slow, count both and stop here.
Otherwise, behave the common way.
My opinion is that you can do whatever you'd like (you're the maintainer
here!) but it'd be nice if the docstring was accurate. If changing the
behavior makes it easier to write a good docstring, that's fine too. Go
with whatever is most useful to you.
--js
Returns dict with the following fields:
'runs': list of test_func results
@@ -47,17 +51,34 @@ def bench_one(test_func, test_env, test_case,
count=5, initial_run=True):
'n-failed': number of failed runs (exists only if at least
one run
failed)
"""
+ runs = []
+ i = 0
if initial_run:
+ t = time.time()
+
print(' #initial run:')
- print(' ', test_func(test_env, test_case))
+ res = test_func(test_env, test_case)
+ print(' ', res)
+
+ if time.time() - t > slow_limit:
+ print(' - initial run is too slow, so it counts')
+ runs.append(res)
+ i = 1
+
+ for i in range(i, count):
+ t = time.time()
- runs = []
- for i in range(count):
print(' #run {}'.format(i+1))
res = test_func(test_env, test_case)
print(' ', res)
runs.append(res)
+ if time.time() - t > slow_limit and len(runs) >= 2:
+ print(' - run is too slow, and we have enough runs,
stop here')
+ break
+
+ count = len(runs)
+
result = {'runs': runs}
succeeded = [r for r in runs if ('seconds' in r or 'iops' in r)]
