bbeaudreault opened a new pull request, #5350: URL: https://github.com/apache/hbase/pull/5350
If a client is not able to read response bytes from the channel faster than the server can serve them, netty's pending outbound bytes will build up. If it builds up long enough, it will result in an OOM. We protect against this with the following: - Use netty's WriteBufferWaterMark to trigger `channel.setAutoRead(false)`. When the high watermark is exceeded, autoRead is disabled. When autoRead is disabled, the server will not continue to read incoming bytes from the client, thus will not enqueue more calls from the client. - Already enqueued calls will still be served, and even with writability checks its possible to OOM. We add a new `fatal` threshold, which should be higher than the high watermark. If exceeded, the connection will be forcibly closed so that we can reclaim the memory. Adds new configs: - hbase.server.netty.writable.watermark.low - hbase.server.netty.writable.watermark.high - hbase.server.netty.writable.watermark.fatal All 3 default to 0. The setAutoRead management is disabled if both `high` and `low` are 0. Enforcing of the fatal threshold is disabled if `fatal` is 0. Thus this new behavior is disabled by default. Adds new metrics: - unwritableTime - histogram showing how much backpressure is being applied due to configured watermark - maxOutboundBytesExceeded - counter showing how many times we needed to forcibly close a connection - nettyTotalPendingOutboundBytes - current aggregate size of pending outbound bytes for all connections on the server - nettyMaxPendingOutboundBytes - current max pending outbound bytes across all channels A version of this has been tested internally at my company. We set the fatal limit to 100mb, and were able to avoid any OOMs in our test case. I still need to add tests to AbstractTestIPC, but wanted to get feedback first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
