Github user PramodSSImmaneni commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/173#issuecomment-159959244
Vlad thanks for the benchmarks for the different sizes. There is consistent
increase in throughput with increase in queue capacities. This warrants a
consideration even if it comes at the cost of higher cpu as we are always
running in a multi-core environment. There may be other mitigation factors we
may not have yet looked at to improve cpu usage.
For a user the higher capacity would translate to a bigger peak memory
size. Why not account for this memory size when we ask for the container. So a
512k capacity, even with a generous 1000 bytes per tuple at full capacity would
be 512mb which is not a lot (we already ask for the same as default for the
buffers that are served). We would add this to the amount of memory we ask for
a container. The user of course would have a choice to trim the queue size down
if they wanted and we could also give them an option to specify this memory
size just like we do for the buffers.
If at different queue capacities a different implementation performs better
we can have a default heuristic strategy that automatically picks the
implementation that is better suited for that range based on our experience
like as you mentioned ArrayBlockingQueue performs better for capacities in the
order of 1k. This will free the user from knowing what to specify and having to
specify it. For power users we could always provide an option to override this
strategy and specify a specific implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
