ege-st commented on code in PR #11496:
URL: https://github.com/apache/pinot/pull/11496#discussion_r1319157318
##########
pinot-core/src/main/java/org/apache/pinot/core/transport/DataTableHandler.java:
##########
@@ -83,5 +98,12 @@ protected void channelRead0(ChannelHandlerContext ctx,
ByteBuf msg) {
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
LOGGER.error("Caught exception while handling response from server: {}",
_serverRoutingInstance, cause);
_brokerMetrics.addMeteredGlobalValue(BrokerMeter.RESPONSE_FETCH_EXCEPTIONS, 1);
+ if (cause instanceof java.lang.OutOfMemoryError) {
Review Comment:
@jasperjiaguo
> In this case the error happens in netty code before we actually go into
channelRead0
This was why we'd been testing having the Broker shutdown, by the time the
event loop can look at the size of the message it's too late. I'd tried to
find some configuration that would set a max size buffer in Netty so that
messages that were too large would simply be dropped before Netty filled all
the direct memory but could not find anything that would achieve that goal.
There's a separate solution we looked at where the Broker would tell the
Server the maximum response message it could safely accept and the Server would
return an error message if the response exceeded that limit. But computing what
that limit would be is complex and we wanted a simpler solution as a first
step. Although I liked this solution because it kept everything as clean as
possible and completely prevented the OOM from happening at all.
Ultimately, I think the right long term solution is to break responses into
a stream of smaller messages and allow the Broker to use backpressure to
prevent OOMs completely. But that's a much bigger lift and having a way to
gracefully deal with the OOM is the right first step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
