FelixYBW commented on PR #7503: URL: https://github.com/apache/incubator-gluten/pull/7503#issuecomment-2412111950
FYI, some more info about jemalloc: Jemalloc: - 5.0 changed a lot. Most document online is for 4.0 and before - Free process: dirty (physical mapped) -> muzzy(MADV_FREE) -> clean (MADV_DONTNEED) - doesn't release virtual memory, just free physical memory - dirty_decay_ms and muzzy_decay_ms control when to release page to OS. Default is 10s. 0 means no delay, -1 means never release - Default 10s is good enough, may increase to 30s - Default background thread is not useful - Tcache size is 32k bydefault, no help when increase to 8M - Madvise(willneed) looks useless to jemalloc allocated memory - Huge allocation memory isn't hold by jemalloc, it will be released (retained) once free is called. oversize_threshold controls the threshold, default is 8M - To cache all memory in pool, use param: muzzy_decay_ms:-1,dirty_decay_ms:-1,background_thread:true,oversize_threshold:5368709121 - dirty_decay_ms=10s doesn't mean the memory block wait until 10s, but keep release in the 10s and release all list by end of 10s. So to keep all page in memory pool, you need to config it as -1  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
