lhotari opened a new pull request, #22585: URL: https://github.com/apache/pulsar/pull/22585
### Motivation There's currently some memory leaks in tests and while investigating the issue, I found out that there's a large number of uncompleted CompletableFutures in the heap dump. Currently the metadata store doesn't complete all pending operations when it is closed. There are multiple problems: - MpscUnboundedArrayQueue.drain will limit the number of entries to 4096 at a time - MpscUnboundedArrayQueue.drain uses relaxedPoll which is eventually consistent - There might be batches in flight, which need to be terminated ### Modifications - replace drain with a while loop in close - replace drain with a for loop in batch flushing - add isClosed checks ### Additional Context CompletableFutures in the heap dump: <img width="1243" alt="image" src="https://github.com/apache/pulsar/assets/66864/c373b65e-c639-4b1d-8ea5-d16a251552b2";> The instances are related to `org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources$$Lambda$1819+0x00007f08a8b65ee8`: <img width="1247" alt="image" src="https://github.com/apache/pulsar/assets/66864/6b289fbd-17e2-4df9-a418-f78770532777";> ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
