Github user 243826 commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/173#issuecomment-159751791
It's not a good idea. It's micro optimization (of not a critical path) at
the cost of complicated implementation for developers and complicated use for
the users.
In the benchmark that Vlad shared starting at 128k queue size we start
seeing that the cost of receiving the event outweighs the cost of processing it
- this is unsustainable practically unless we are doing next to none processing
of the event. I even suspect that it's the same event that's being pushed as
opposed to pushing a new event every time through SPSC (Vlad - do you have
reuse_tuple or equivalent flag turned on?). Practically a long before we reach
that throughput with larger queue size - we breach through the barrier between
the ContainerLocal and ThreadLocal. The timing is driven by the processing
complexity of subscriber operator. At that point you may as well switch to
ThreadLocal and enjoy much more and optimal throughput (the benchmark for
ThreadLocal it also very academic).
-
Chetan
From: Thomas Weise
<[email protected]<mailto:[email protected]>>
Reply-To: apache/incubator-apex-core
<[email protected]<mailto:[email protected]>>
Date: Wednesday, November 25, 2015 at 12:54 PM
To: apache/incubator-apex-core
<[email protected]<mailto:[email protected]>>
Cc: Chetan Narsude <[email protected]<mailto:[email protected]>>
Subject: Re: [incubator-apex-core] APEX-254 & APEX-269 (#173)
Vlad, this is great data. How about selecting the default reservoir
implementation based on the queue size? ArrayBlockingQueueReservoir for small
queue size, spsc for larger size? And leave the final say about the
implementation to the user?
-
Reply to this email directly or view it on
GitHub<https://github.com/apache/incubator-apex-core/pull/173#issuecomment-159729354>.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
