dlg99 commented on a change in pull request #1088: ISSUE #1086 (@bug
W-4146427@) Client-side backpressure in netty (Fixes:
io.netty.util.internal.OutOfDirectMemoryError under continuous heavy load)
URL: https://github.com/apache/bookkeeper/pull/1088#discussion_r173341459
##########
File path:
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java
##########
@@ -437,7 +438,7 @@ public ClientConfiguration
setClientWriteBufferLowWaterMark(int waterMark) {
* @return netty channel write buffer high water mark.
*/
public int getClientWriteBufferHighWaterMark() {
- return getInt(CLIENT_WRITEBUFFER_HIGH_WATER_MARK, 64 * 1024);
+ return getInt(CLIENT_WRITEBUFFER_HIGH_WATER_MARK, 512 * 1024);
Review comment:
More interesting question is why 64K.
Without backpressure that value is simply ignored. netty would switch
isWritable on and off but won't prevent one from writing to the channel.
With backpressure I've tested effect of different values of HWM (chart
attached to the CR) and throughput with backpressure and default HWM of 64K was
way too low. Most of the entries in my test were 64KB.
Also from what I read, channel allocated whole chunk from
PooledbyteBufAllocator to not deal with concurrent access.
"The PooledbyteBufAllocator allocates 16 MiB per chunk by default. If all
chunks are full or the pool was not able to find a room for the requested
buffer, it will allocate another chunk. If all buffers allocated from the
chunk are released, the chunk will be deallocated automatically."
https://groups.google.com/forum/#!searchin/netty/PooledByteBufAllocator$20chunk%7Csort:date/netty/3riW1A26QKc/5pjs-n-tuHYJ
As far as I can tell this still stands.
So HWM < chunk size makes sense to reduce number of submitted requests that
netty won't be able to write out anyway if server is not reading. But larger
HWM helps with throughput when bookies are healthy / accommodates spikes in
load better.
So I decided to increase default to safe (less than chunk size) value that
nevertheless improves throughput in case of backpressure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
