BenoitHanotte opened a new pull request #9265: [FLINK-13477] Add `memory-overhead-ratio` conf for containers URL: https://github.com/apache/flink/pull/9265 ## What is the purpose of the change Currently, the `-XX:MaxDirectMemorySize` parameter is set as: MaxDirectMemorySize = containerMemoryMB - heapSizeMB However as explained at https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html, `MaxDirectMemorySize` only sets the maximum amount of memory that can be used for direct buffers, thus the amount of off-heap memory used can be greater than that value, leading to the sum of heap + off-heap memory used being higher than the allocated container memory and the container being killed by Yarn or Mesos for that reason. In addition, users might want to allocate off-heap memory through native code, in which case they will want to keep some of the container memory free and unallocated by Flink. ## Brief change log - *Add `containerized.memory-overhead-ratio` config to allow keeping some of the container memory free (default is 0 to keep the current behavior)* ## Verifying this change This change added tests and can be verified as follows: - *Added `testUsableMemory` unit test to ensure memory-overhead-ratio is honored in `ContainerTaskManagerParametersTest.java`* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
