Hi, Stepan!
I looked at the benchmark code and the overall methodics, discussed it
with fellow programmers, and came up with basically two remarks.
First of all, I think, the key for the good (or, at least,
unobjectionable) measurement is to isolate the object being measured
from the influence of the measurement tool. The usual pattern we use in
Python looks like:
```
from time import time
bench_start = time()
for _ in range(number_of_tests):
do_test()
bench_end = time()
print('Performance is {} tests per second'.format(
number_of_tests / (bench_end - bench_start)
))
```
I think, you got the idea: the measurement consists almost solely of the
time taken by our subject function `do_test`. As few other code as
possible influence the result.
Now, let's take a look at your code:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Ideally, the `while` loop should include only `cache.put(last_key,
some_precomputed_value)`. But instead it also includes:
- a series of the `time()` calls, which could be mostly excluded from
the measured time, if the measurement done right; each call is probably
addresses the HPET device, or network time, or both,
- some floating-point calculations, including `round()`, which is hardly
necessary,
- formatting and output of the intermediate result.
I suppose the measurement influence can be quite significant here, but
it is at least more or less constant for each test.
But if we look at the other benchmarks:
https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
[Node.js]
https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce [Java]
The extra code that influence the measurement is not equivalent across
all platforms. For example, PHP's `time()` is most probably lighter than
Python `time()`, since it do not give out milliseconds and may address
RTC, not HPET. So the platform-to-platform comparison in your benchmark
does not look completely fair to me.
The second remark concerns not the measurement procedure, but the object
being measured.
The only client operation being used is OP_CACHE_PUT with a payload of a
primitive type. (BTW the type is `Long` in case of the Python client;
what about the other clients? Is it `Int`?) I afraid that such an
object, even being properly isolated from the measurement tool, would
show mostly the throughput of the underlying platform's sockets
implementation, not the performance of the client's code. To show the
potential of the thin client itself, more varied and fancy tasks needed,
i. e. serialization/deserialization of the Complex objects.
But it depends on the goal of the benchmarking. If showing off the raw
platform agility was intended, than this objection is removed.
Dmitry
On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
Hi, all
Tried to compare new thin client performance and made similar benchmarks for
each one
And result not so good for python
Ticket with results and bench code:
https://issues.apache.org/jira/browse/IGNITE-9824
Py code src:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
Dmitry, please review results and bench code, maybe somthing wrong or it's
expected numbers?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
