Github user vrozov commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/173#issuecomment-159438298
I don't agree that it is not possible to use 512K queue size. Depending on
tuple size, memory availability and processing requirements queue size of 512K
or larger may be used by applications. Whether or not default queue size should
be changed to larger value is discussable. Based on my experience and what I've
heard, there is no good default value for the queue size and applications need
to choose between slightly larger memory consumption or a performance boost.
Note that 4x performance improvement of SpscArrayQueueReservoir over
CircularBufferReservoir is for CircularBufferReservoir configured also for 512K
queue size. With the default queue size, CircularBufferReservoir scales only to
3 million events per second.
It is not clear why it is necessary to have exact size() and where do we
pay price for loosing accuracy of the size(). Performance benchmark does not
show any degradation for the CircularBufferReservoir that does not use size()
and uses BlockingQueue interface.
While benchmark is done under ideal conditions, it measures scalability of
the Apex framework. SpscArrayQueueReservoir provides the infrastructure that is
more scalable compared to CircularBufferReservoir and
ArrayBlockingQueueReservoir. IMO, this should be sufficient to justify pulling
the changes in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
