[ https://issues.apache.org/jira/browse/FLINK-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216585#comment-16216585 ]
ASF GitHub Bot commented on FLINK-7400: --------------------------------------- Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/4506#discussion_r146479627 --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java --- @@ -132,6 +132,36 @@ public void perJobYarnCluster() { } /** + * Test per-job yarn cluster and memory calculations for off-heap use (see FLINK-7400). + * + * <p>This also tests the prefixed CliFrontend options for the YARN case + * We also test if the requested parallelism of 2 is passed through. + * The parallelism is requested at the YARN client (-ys). + */ + @Test + public void perJobYarnClusterOffHeap() { + LOG.info("Starting perJobYarnCluster()"); + addTestAppender(JobClient.class, Level.INFO); + File exampleJarLocation = new File("target/programs/BatchWordCount.jar"); + Assert.assertNotNull("Could not find wordcount jar", exampleJarLocation); + runWithArgs(new String[]{"run", "-m", "yarn-cluster", + "-yj", flinkUberjar.getAbsolutePath(), "-yt", flinkLibFolder.getAbsolutePath(), + "-yn", "1", + "-ys", "2", //test that the job is executed with a DOP of 2 + "-yjm", "768", + "-ytm", "1024", + "-yD", "taskmanager.memory.off-heap=true", + "-yD", "taskmanager.memory.size=246", // this should fit! + "-yD", "taskmanager.memory.preallocate=true", exampleJarLocation.getAbsolutePath()}, + /* test succeeded after this string */ + "Job execution complete", + /* prohibited strings: (we want to see "DataSink (...) (2/2) switched to FINISHED") */ + new String[]{"DataSink \\(.*\\) \\(1/1\\) switched to FINISHED"}, --- End diff -- The comment above says something different than this line `(1/1)` vs. `(2/2)` > off-heap limits set to conservatively in cluster environments > ------------------------------------------------------------- > > Key: FLINK-7400 > URL: https://issues.apache.org/jira/browse/FLINK-7400 > Project: Flink > Issue Type: Bug > Components: Cluster Management, Mesos, YARN > Affects Versions: 1.3.0, 1.3.1, 1.3.2 > Reporter: Nico Kruber > Assignee: Nico Kruber > > Inside {{ContaineredTaskManagerParameters}}, since FLINK-6217, the > {{offHeapSize}} is set to the amount of memory Flink will use off-heap which > will be set as the value for {{-XX:MaxDirectMemorySize}} in various cases. > This does not account for any off-heap use by other components than Flink, > e.g. RocksDB, other libraries, or the JVM itself. > We should add the {{cutoff}} from the {{CONTAINERIZED_HEAP_CUTOFF_RATIO}} > configuration parameter to {{offHeapSize}} as implied by the description on > what this parameter is there for. -- This message was sent by Atlassian JIRA (v6.4.14#64029)