lhotari commented on PR #22494: URL: https://github.com/apache/pulsar/pull/22494#issuecomment-2053617569
> > One interesting detail is that in the load test, the system couldn't keep up with the load without making the optimization to direct buffer allocation in commit [eb53342](https://github.com/apache/pulsar/commit/eb53342e6a73788e470a1bb13c9722ccb25cc584). > > I guess it is related to CompositeByteBuf#consolidateIfNeeded, once the number of components exceeds MAX_COMPONENT, it will combine all the components to one component, memory copy happens. Memory copying isn't the biggest problem. A bigger problem is direct memory OOM that seems to happen when the memory space is so fragmented that there isn't free space for the allocation requests. The Netty pool has a maximum chunk size (8MB default) and all larger allocations are huge allocations that aren't pooled. That's why I think the added logic helps since it pre-allocates up to 8MB chunks. Memory copying will happen, but that seems to be fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
