gaborgsomogyi commented on PR #28427: URL: https://github.com/apache/flink/pull/28427#issuecomment-5283713644
**TaskManager never records S3 filesystem metrics — `attachMetrics()` runs after the S3 filesystem is already cached** `ClusterEntrypoint.java` intentionally moves `attachMetrics()` early, with a comment explaining why: ```java // Attach file system metrics before HA/blob services can create cached clients. FileSystem.attachMetrics(processMetricGroup); ``` `TaskManagerRunner.java` doesn't get the same treatment — `FileSystem.attachMetrics(taskManagerMetricGroup.f0)` is inserted inside `startTaskManager()`, but something earlier in TM startup already resolves the checkpoint S3 path and caches the filesystem instance first. Since `FileSystem`'s static cache holds instances for the process lifetime, and the metric publisher is baked in immutably at creation time, that one early instance never gets metrics — and neither does anything reusing it afterward. Reproduced on a live deployment: TM log shows the S3 filesystem created at `13:30:43`, while `startTaskManager()` (where `attachMetrics()` lives) doesn't start until `13:30:46`. Result: 89 completed checkpoints, hundreds of GB uploaded, zero `filesystem.*` metrics ever registered on that TM — while JM (correct ordering) showed metrics fine for the same job. Suggest applying the same fix pattern to `TaskManagerRunner` as `ClusterEntrypoint`: move `attachMetrics()` before whatever first touches the checkpoint/state filesystem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
