zhengchenyu opened a new issue, #2116: URL: https://github.com/apache/incubator-uniffle/issues/2116
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues. ### What would you like to be improved? Initially, under k8s deployment, that is jdk11, I found used_direct_memory_size was always -2. I found debug log `[main] [DEBUG] PlatformDependent0.<clinit> - direct buffer constructor: unavailable`. It means we will use DirectByteBuffer constructor with cleaner, then means that `Netty will not enforce max memory, and instead will defer to JDK.` I search the code, found that netty have two ways: ## (1) use jdk cleaner * Netty will not limit the memory. * We should use `-XX:MaxDirectMemorySize` to limit DirectByteBuffer. If the limit is reached, GC is triggered. * We should use metric: `jvm_buffer_pool_used_bytes{pool="direct"}` ## (2) do not use jdk cleaner * Netty will limit memory with `io.netty.maxDirectMemory`. If the limit is reached, will throw exception. * We can also use `-XX:MaxDirectMemorySize` to limit DirectByteBuffer, but the usage should be very low. I think there are no need to limit. * We should use metrics: `used_direct_memory_size` > BTW, the native memory used in FileOuputStream can not be limited and monitored. We should avoid that. ### How should we improve? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
