Charles E. Rolke created DISPATCH-2000:
------------------------------------------
Summary: Consider increasing default internal buffer size
Key: DISPATCH-2000
URL: https://issues.apache.org/jira/browse/DISPATCH-2000
Project: Qpid Dispatch
Issue Type: Improvement
Components: Router Node
Affects Versions: 1.15.0
Reporter: Charles E. Rolke
Tests with the TCP adaptor and a 4k-byte buffer size show _significant
throughput increases_ with _otherwise minimal code changes_. For this
discussion, assume that the new buffer size will be 4k bytes.
Note: This discussion nearly parallels the discussion of increasing hard drive
sector size to 4K. See [Transition to Advanced Format 4K Sector Hard Drives
|[https://www.seagate.com/tech-insights/advanced-format-4k-sector-hard-drives-master-ti/].]
The pros and cons of increasing the size for disk drives and for qpid-dispatch
are pretty much the same.
Dispatch has never had anything but 512-byte internal buffers. This lends a lot
of weight for keeping the current setting. However, from day one the self tests
exercise various buffer sizes from 1-byte to 10000-bytes and the tests have
worked fine. This is a positive indication that physically the buffer functions
and buffer management proper would work with any buffer size.
Logically there are some other issues. These must be understood and considered
before moving forward with any change.
* Expect total memory usage to go up due to unused buffer space with small
messages
* Expect total memory usage to go down due to more efficient buffer use with
large messages
* Policy implements Q2 throttling with a buffer count limit. If that limit is
not adjusted then flow control will not take effect until 8x as much AMQP
session data has been buffered.
Adjusting the buffer size from 512 to 4096 must happen either at compile time
or at run time very early, before qd_router_setup_late() is called, when
buffers start being created and buffer size cannot be changed any more. A pull
request is in the works that adds a --4k-buffer switch to the command line to
simplify testing with the larger buffer size.
I expect that going to 4K buffers will have a performance improvement certainly
for the TCP adaptor but also for the other protocol adaptors and for AMQP as
well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
