xintongsong commented on a change in pull request #10608:
[FLINK-15300][Runtime] Fix sanity check to not fail if shuffle memory fraction
is out of min/max range
URL: https://github.com/apache/flink/pull/10608#discussion_r359138858
##########
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/TaskExecutorResourceUtilsTest.java
##########
@@ -198,6 +198,35 @@ public void testConfigShuffleMemoryRange() {
});
}
+ @Test
+ public void testSanityCheckShuffleMemoryFractionLessThanMin() {
+ testSanityCheckShuffleMemoryFractionOutOfMinMaxRange(true);
+ }
+
+ @Test
+ public void testSanityCheckShuffleMemoryFractionGreaterThanMax() {
+ testSanityCheckShuffleMemoryFractionOutOfMinMaxRange(false);
+ }
+
+ private void
testSanityCheckShuffleMemoryFractionOutOfMinMaxRange(boolean
isShuffleMemoryLessThanMin) {
+ final MemorySize shuffleSize = MemorySize.parse("64m");
+ float shuffleMemFraction = isShuffleMemoryLessThanMin ? 0.1f :
0.3f;
+
+ // configure task heap, managed and shuffle memory which add up
to the configured total Flink memory
+ Configuration conf = new Configuration();
+ conf.setString(TaskManagerOptions.SHUFFLE_MEMORY_MIN,
shuffleSize.getMebiBytes() + "m");
+ conf.setString(TaskManagerOptions.SHUFFLE_MEMORY_MAX,
shuffleSize.getMebiBytes() + "m");
+ conf.setFloat(TaskManagerOptions.SHUFFLE_MEMORY_FRACTION,
shuffleMemFraction);
+ conf.setString(TaskManagerOptions.TOTAL_FLINK_MEMORY, "350m");
+ conf.setString(TaskManagerOptions.TASK_HEAP_MEMORY, "2097150b");
+ conf.setString(TaskManagerOptions.MANAGED_MEMORY_SIZE,
"146800642b");
+ conf.setString(TaskManagerOptions.FRAMEWORK_HEAP_MEMORY, "16m");
Review comment:
I also think these three options are irrelevant in this test case.
I understand that without these three lines of codes, the memory sizes do
not add up with the default values. That can be easily fixed by setting larger
total flink / shuffle size. E.g., 1000m total flink and 200m shuffle size
should work.
And it also provides better readability, because it's easier to tell that
200m is 0.2 of 1000m (which is smaller than 0.3 and later than 0.1), compared
to 64m is ~0.18 of 350m.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
