azagrebin opened a new pull request #11109: [FLINK-15758][MemManager] Release segment and its unsafe memory in GC Cleaner URL: https://github.com/apache/flink/pull/11109 ## What is the purpose of the change After #9747, managed memory is allocated from UNSAFE, not as direct nio buffers as before 1.10. The releasing of segments released also underlying unsafe memory which is dangerous in general as there can be still references to java objects giving access to the released memory. If this reference ever leaks, the illegal memory access can result in memory corruption of other code parts w/o even segmentation fault. The solution can be similar to how Java handles direct memory limit: - underlying byte buffers of segments are registered to phantom reference queue with a Java GC cleaner which releases the unsafe memory - all allocations and reservations are managed by a memory limit and an atomic available memory - if available memory is not enough while reserving, the phantom reference queue processing is triggered to run hopefully discovered by GC cleaners - memory can be released directly or in a GC cleaner The GC is also sped up by nulling out byte buffer reference in `HybridMemorySegment#free` which is inaccessible anyways after freeing. Otherwise also a lot of tests, which hold accidental references to memory segments, have to be fixed to not hold them. The `MemoryManager#verifyEmpty` checks that everything can be GC'ed at the end of the tests and after slot closing in production to detect memory leaks if any other references are held, e.g. from `HybridMemorySegment#wrap`. ## Brief change log - Simply MemoryManager by removing KeyedBudgetManager as the managed memory is only on-heap now - Remove MemoryManager#AllocationRequest - Implement custom unsafe memory control, similar to what JVM does to control direct memory limit - Use AtomicLong to control allocated memory and optimistic allocation retry (like in nio.Bits#tryReserveMemory) and trigger GC cleaners if memory is unavailable - adjust and add unit tests ## Verifying this change unit tests
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
