xintongsong opened a new pull request #9801: [FLINK-13983][runtime] Launch task executor with new memory calculation logics URL: https://github.com/apache/flink/pull/9801 ## What is the purpose of the change This PR is part of FLIP-49. It uses the new memory configuration and calculation logics for launching task executors. This PR is not yet accomplished. - TM should decide sizes of managed memory pools with new configuration and calculation logics. This is currently blocked on #9693. - Changes to the Mesos deployment need to be manually verified. This PR is based on #9760. ## Brief change log - c16e4e4..870b4a6: Commits of previous PR #9760. - 4389594: Introduce `BashJavaUtils`, which allows to call java implemented utils from bash scripts. - ecff467: Refactor `TaskManagerHeapSizeCalculationJavaBashTest`, extract common `JavaBashTestBase` for executing testing bash scripts from java test cases. - accef46: Add `BashJavaUtilsTest` to validate calling `BashJavaUtils` from bash scripts. - 0e02ef3: TM startup bash scripts use new memory configuration and calculation logics to launch task executors (when flip49 is enabled) - 8a62b0b: Fix dynamic configuration formats (there should be a space between `'-D'`and `'<key>=<value>'`) - ff5b85c: Remove heading/trailing/duplicated whitespaces from shell command generated by `BootstrapTools`. This simplifies relevant test cases from carefully dealing with white space counts. - 71579c5: `YarnResourceManager` launches task executors with new memory configuration and calculation logics (when flip49 is enabled) - 7ff8ca5: `MesosResourceManager` launches task executors with new memory configuration and calculation logics (when flip49 is enabled) - 3ea28aa: TM decide memory size of `ShuffleEnvironment` with new memory configuration and calculation logics (when flip49 is enabled) ## Verifying this change - Add `BashJavaUtilsTest` to creating dynamic configs and JVM parameters by calling java codes from bash scripts. - Manually test launching task executors with proper dynamic configs and JVM parameters when using FLIP-49 new configurations. - Add `BootstrapTools#testGetTaskManagerShellCommand` to test generating task executor launching command with proper dynamic configs and JVM parameters on Yarn. - **NOT YET**: Manually test launching task executors with proper dynamic configs and JVM parameters when using FLIP-49 new configurations. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
