gaozhangmin opened a new issue, #3737:
URL: https://github.com/apache/bookkeeper/issues/3737
**BUG REPORT**
Version: 4.15.3
Recently, in our product, we found that some bookies direct memory increases
suddenly, But never goes down.
We found that the bookie has too many add request on it, and it goes through
add request throttle.
The Bookie logs :
```
INFO org.apache.bookkeeper.proto.BookieRequestProcessor - Too many add
requests in progress, disabling autoread on channel [id: 0xe4a16245, L:/x:3181
- R:/x:43326]
INFO org.apache.bookkeeper.proto.BookieRequestProcessor - Re-enabled
autoread on channel [id: 0xc6c25478, L:/x:3181 - R:/x:43738] after AddRequest
delay of 32048290371 nanos
```
The Broker logs showed entries wrote timeout
```
2023-01-14 08:32:22.367 [BookKeeperClientWorker-OrderedExecutor-21-0] WARN
org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (187083, 0)
to bookie (0, x.x.x.x:3181): Bookie operation timeout
```
From bookie memory dump, We found there are many ParseAddRequest object
stayed alive even this bookie had been set to readonly for a long time.
I think the problem lies in When add request is going to be processed.
Firstly, WriteEntryProcessor is created
https://github.com/apache/bookkeeper/blob/9e8cda30d4e63371ff9ae12e64189aee47ad8000/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/WriteEntryProcessor.java#L51-L57
The Line 55 `requestProcessor.onAddRequestStart(channel)` will perform add
request throttle if enabled, the throttle here would cause client disconnect
after timeout. but the ParseAddRequest object is never released.
That's where leak happened.
***To Reproduce***
1. Enable add request throttle, set a small value maxAddsInProgressLimit=1
2. Start produce data to bookie.
3. slowly, you would find the direct memory increased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
