[
https://issues.apache.org/jira/browse/RATIS-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Pal updated RATIS-2632:
--------------------------------
Description:
When the leader streams a snapshot to a lagging follower, the chunk send loop
in
[GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783]
calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever
checking `*CallStreamObserver.isReady()*`.
The append-entries path already [checks
isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366],
but the snapshot path does not, so it is possible that chunks are enqueued
faster than the network can drain them.
On the default path this is mitigated by the
*INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT* being 8, but since this can be
set to 0, in the worst case the StreamObserverWithTimeout's *ResourceSemaphore*
is null and cause:
- Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue
when the follower is slow.
- Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which
can crash the leader and impact the whole group.
was:
When the leader streams a snapshot to a lagging follower, the chunk send loop
in
[GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783]
calls `*snapshotRequestObserver.onNext(request)*` for every chunk without ever
checking `*CallStreamObserver.isReady()*`.
The append-entries path already [checks
isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366],
but the snapshot path does not, so it is possible that chunks are enqueued
faster than the network can drain them.
This can cause:
- Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write queue
when the follower is slow.
- Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which
can crash the leader and impact the whole group.
> install-snapshot chunk loop send lacks isReady() backpressure
> -------------------------------------------------------------
>
> Key: RATIS-2632
> URL: https://issues.apache.org/jira/browse/RATIS-2632
> Project: Ratis
> Issue Type: Improvement
> Components: gRPC
> Reporter: Abhishek Pal
> Assignee: Abhishek Pal
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When the leader streams a snapshot to a lagging follower, the chunk send loop
> in
> [GrpcLogAppender.installSnapshot()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L775-L783]
> calls `*snapshotRequestObserver.onNext(request)*` for every chunk without
> ever checking `*CallStreamObserver.isReady()*`.
> The append-entries path already [checks
> isReady()|https://github.com/apache/ratis/blob/a1bbf47f98a83d3bbddb4357c8a9b96bcfd7c727/ratis-grpc/src/main/java/org/apache/ratis/grpc/server/GrpcLogAppender.java#L366],
> but the snapshot path does not, so it is possible that chunks are enqueued
> faster than the network can drain them.
> On the default path this is mitigated by the
> *INSTALL_SNAPSHOT_REQUEST_ELEMENT_LIMIT_DEFAULT* being 8, but since this can
> be set to 0, in the worst case the StreamObserverWithTimeout's
> *ResourceSemaphore* is null and cause:
> - Unbounded outbound buffering of snapshot chunks in the gRPC/Netty write
> queue when the follower is slow.
> - Leader OOM / direct-memory exhaustion during large-snapshot catch-up, which
> can crash the leader and impact the whole group.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
