[
https://issues.apache.org/jira/browse/FLINK-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216584#comment-16216584
]
ASF GitHub Bot commented on FLINK-7400:
---------------------------------------
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4506#discussion_r146479686
--- Diff:
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java
---
@@ -132,6 +132,36 @@ public void perJobYarnCluster() {
}
/**
+ * Test per-job yarn cluster and memory calculations for off-heap use
(see FLINK-7400).
+ *
+ * <p>This also tests the prefixed CliFrontend options for the YARN case
+ * We also test if the requested parallelism of 2 is passed through.
+ * The parallelism is requested at the YARN client (-ys).
+ */
+ @Test
+ public void perJobYarnClusterOffHeap() {
+ LOG.info("Starting perJobYarnCluster()");
+ addTestAppender(JobClient.class, Level.INFO);
+ File exampleJarLocation = new
File("target/programs/BatchWordCount.jar");
+ Assert.assertNotNull("Could not find wordcount jar",
exampleJarLocation);
+ runWithArgs(new String[]{"run", "-m", "yarn-cluster",
+ "-yj", flinkUberjar.getAbsolutePath(), "-yt",
flinkLibFolder.getAbsolutePath(),
+ "-yn", "1",
+ "-ys", "2", //test that the job is executed
with a DOP of 2
+ "-yjm", "768",
+ "-ytm", "1024",
+ "-yD", "taskmanager.memory.off-heap=true",
+ "-yD", "taskmanager.memory.size=246", // this
should fit!
--- End diff --
Could you add some explanation why this fits now and not before?
> off-heap limits set to conservatively in cluster environments
> -------------------------------------------------------------
>
> Key: FLINK-7400
> URL: https://issues.apache.org/jira/browse/FLINK-7400
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management, Mesos, YARN
> Affects Versions: 1.3.0, 1.3.1, 1.3.2
> Reporter: Nico Kruber
> Assignee: Nico Kruber
>
> Inside {{ContaineredTaskManagerParameters}}, since FLINK-6217, the
> {{offHeapSize}} is set to the amount of memory Flink will use off-heap which
> will be set as the value for {{-XX:MaxDirectMemorySize}} in various cases.
> This does not account for any off-heap use by other components than Flink,
> e.g. RocksDB, other libraries, or the JVM itself.
> We should add the {{cutoff}} from the {{CONTAINERIZED_HEAP_CUTOFF_RATIO}}
> configuration parameter to {{offHeapSize}} as implied by the description on
> what this parameter is there for.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
