AntiTopQuark commented on issue #2284: URL: https://github.com/apache/kvrocks/issues/2284#issuecomment-2158599483
> Hello @AntiTopQuark, I am also interested in this issue. When I implemented the Sort command before, there was a need for memory restrictions. > > ``` > /// SORT_LENGTH_LIMIT limits the number of elements to be sorted > /// to avoid using too much memory and causing system crashes. > /// TODO: Expect to expand or eliminate SORT_LENGTH_LIMIT > /// through better mechanisms such as memory restriction logic. > constexpr uint64_t SORT_LENGTH_LIMIT = 512; > ``` > > It seems that the current memory statistics of kvrocks are still coarse-grained process-level statistics. Do we need more fine-grained tracking statistics? I've collected some references: https://cwiki.apache.org/confluence/display/DORIS/DSIP-002%3A+Refactor+memory+tracker+on+BE https://doris.apache.org/blog/Say-Goodbye-to-OOM-Crashes/ https://www.modb.pro/db/1798912145290776576 > > They seem to utilize the memory allocator to achieve this, would you like to share and discuss your thoughts? I briefly read through the three articles and found that most methods are quite similar. They all involve tracking the type, size, and frequency of allocations, as well as the stack at the time of allocation, to avoid OOM errors and to diagnose memory leaks. I am more familiar with the OceanBase database, which mainly implements a series of ObAllocator's `alloc` functions. These functions track the size and address of allocations (with address tracking enabled via a switch), and compare them against tenant-level memory limits to prevent OOM. If memory allocation fails, an error code is returned to the client. You can find more information at the following links: [OceanBase Official Documentation](https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000819454) [OceanBase GitHub Code](https://github.com/oceanbase/oceanbase/blob/c51018527616b6da76674108d20bad8a7bff6431/deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp#L424) For the issue with kvrocks, I think it’s best to keep it simple. Implementing overrides for malloc and new to record the number of allocations and the amount of memory should suffice. Additionally, you can limit memory allocation using the max-memory configuration item and set a limit on the maximum number of connections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
