Hi Dmitry,
I agree with your comments on benchmarking code. As more fair alternative
we may measure time of loading N elements into the cache, so that it will
be necessary to call time() only twice. However, provided that we have real
network here, and according to numbers one PUT in Python client requires
about 0.3 msec, I hardly can imagine that call to time() or round() may
dominate in that response time. But I said, we can easily rule that out by
slight benchmark rewrite.
As far as serialization, I would prefer to keep primitive objects at the
moment, because AFAIK the purpose of the benchmark was to assess underlying
infrastructure, looking for some obvious performance issues. Every platform
have sockets which use more or less the same API of underlying OS. To the
contrast, performance of serialization mechanics may differ widely between
platforms depending on implementation. E.g. in Java we spend a lot of time
on fine-grained tuning, applied a lot of speculative optimizations. Add to
this JIT nature of Java, and you will hardly ever beat Java serialization
engine from any interpreted language. So primitive data types make good
sense to me.
At this point our goal is not make Python equally fast with other
platforms, but rather to understand why is it slower than others. Ignite is
a product which brings speed to end users. If they do no have speed, they
will not use Ignite. So performance questions are always of great
importance for us.
Vladimir.
On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:
> Hi, Stepan!
>
> I looked at the benchmark code and the overall methodics, discussed it
> with fellow programmers, and came up with basically two remarks.
>
> First of all, I think, the key for the good (or, at least,
> unobjectionable) measurement is to isolate the object being measured
> from the influence of the measurement tool. The usual pattern we use in
> Python looks like:
>
> ```
> from time import time
>
>
> bench_start = time()
>
> for _ in range(number_of_tests):
> do_test()
>
> bench_end = time()
>
> print('Performance is {} tests per second'.format(
> number_of_tests / (bench_end - bench_start)
> ))
> ```
>
> I think, you got the idea: the measurement consists almost solely of the
> time taken by our subject function `do_test`. As few other code as
> possible influence the result.
>
> Now, let's take a look at your code:
>
> https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
>
> Ideally, the `while` loop should include only `cache.put(last_key,
> some_precomputed_value)`. But instead it also includes:
>
> - a series of the `time()` calls, which could be mostly excluded from
> the measured time, if the measurement done right; each call is probably
> addresses the HPET device, or network time, or both,
>
> - some floating-point calculations, including `round()`, which is hardly
> necessary,
>
> - formatting and output of the intermediate result.
>
> I suppose the measurement influence can be quite significant here, but
> it is at least more or less constant for each test.
>
> But if we look at the other benchmarks:
>
> https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
> [Node.js]
> https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
> https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
> [Java]
>
> The extra code that influence the measurement is not equivalent across
> all platforms. For example, PHP's `time()` is most probably lighter than
> Python `time()`, since it do not give out milliseconds and may address
> RTC, not HPET. So the platform-to-platform comparison in your benchmark
> does not look completely fair to me.
>
> The second remark concerns not the measurement procedure, but the object
> being measured.
>
> The only client operation being used is OP_CACHE_PUT with a payload of a
> primitive type. (BTW the type is `Long` in case of the Python client;
> what about the other clients? Is it `Int`?) I afraid that such an
> object, even being properly isolated from the measurement tool, would
> show mostly the throughput of the underlying platform's sockets
> implementation, not the performance of the client's code. To show the
> potential of the thin client itself, more varied and fancy tasks needed,
> i. e. serialization/deserialization of the Complex objects.
>
> But it depends on the goal of the benchmarking. If showing off the raw
> platform agility was intended, than this objection is removed.
>
> Dmitry
>
> On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:
> > Hi, all
> >
> > Tried to compare new thin client performance and made similar benchmarks
> for
> > each one
> > And result not so good for python
> >
> > Ticket with results and bench code:
> > https://issues.apache.org/jira/browse/IGNITE-9824
> >
> > Py code src:
> > https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb
> >
> > Dmitry, please review results and bench code, maybe somthing wrong or
> it's
> > expected numbers?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
>
