XComp opened a new pull request #17176: URL: https://github.com/apache/flink/pull/17176
Backport for #16989 Copied from original PR: ## What is the purpose of the change We struggled to investigate [FLINK-23611](https://issues.apache.org/jira/browse/FLINK-23611) due to missing Flink logs. It appears that something went wrong with the Flink cluster. The stop signal wasn't retrieved by the YARN Session Cluster thread for some reason which made the test wait forever for the thread to finish. The tests are executed on AzureCI through `tools/ci/test_controller.sh` which implements a watchdog mechanism that checks the logs (`stdout` and `mvn-*.log`) for new content and kills the test if there's no output for a given amount of time (900s). YARN does produce regular logs, though, through the `org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService` logger. These log messages are generated every 10 minutes: ``` 22:51:31,785 [AsyncDispatcher event handler] INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService [] - Cache Size Before Clean: 0, Total Deleted: 0, Public Deleted: 0, Private Deleted: 0 22:51:32,398 [AsyncDispatcher event handler] INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService [] - Cache Size Before Clean: 0, Total Deleted: 0, Public Deleted: 0, Private Deleted: 0 ``` Hence, the test is not killed while waiting for the YARN cluster to finish. The Flink logs, as a consequence, are not copied over from the YARN application folder into the build artifact folder as part of the [tools/ci/test_controller.sh](https://github.com/apache/flink/blob/master/tools/ci/test_controller.sh#L128) execution. ## Brief change log - Hotfix: Fixes bug in path creation - Hotfix: Improves local log4j configuration for YARN tests - Disables INFO logs for `org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService` in ci log4j configuration ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
