GitHub user NicoK opened a pull request:

    [FLINK-7400][cluster] fix off-heap limits set to conservatively in cluster 

    ## What is the purpose of the change
    Inside `ContaineredTaskManagerParameters`, since #3648, the `offHeapSize` 
is set to the amount of memory Flink will use off-heap which will be set as the 
value for `-XX:MaxDirectMemorySize` in various cases, e.g. YARN or Mesos. This 
does not account for any off-heap use by other components than Flink, e.g. 
RocksDB, other libraries, or the JVM itself.
    Please note that this affects at least all batch programs with the 
following options set (which do not make much sense for streaming):
    taskmanager.memory.size=<any value>
    If, instead, `taskmanager.memory.fraction` is used, programs may be safe 
due to but the actual 
additional buffer that we get from that may be too small, especially if RocksDB 
or other libraries using off-heap memory are used.
    This PR adds the `cutoff` from the 
`containerized.heap-cutoff-ratio`/`containerized.heap-cutoff-min` configuration 
parameters to `offHeapSize` as implied by the description of these two options.
    ## Brief change log
    - include the cut-off memory (removed from the container memory size for 
further calculations) into the off-heap part
    - add a unit test verifying the bug fix in a YARN environment
    ## Verifying this change
    This change added tests and can be verified as follows:
    - added `YARNSessionCapacitySchedulerITCase#perJobYarnClusterOffHeap()` 
test that validates that we have enough memory available and the bounds are not 
too strict
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes: memory calculations)
    ## Documentation
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (JavaDocs)

You can merge this pull request into a Git repository by running:

    $ git pull flink-7400

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4506
commit 60d40cde20686b4b1b2d15dc838b15ed0cd994cc
Author: Nico Kruber <>
Date:   2017-08-09T09:53:03Z

    [FLINK-7400][cluster] fix cut-off memory not used for off-heap reserve as 
    + fix description of `containerized.heap-cutoff-ratio`

commit 4135a223288608444d324da333cfdd70117c796d
Author: Nico Kruber <>
Date:   2017-08-09T14:16:31Z

    [FLINK-7400][yarn] add an integration test for yarn container memory 
restrictions using off-heap memory


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

Reply via email to