Another possibility is that the CGroup mechanism is not enabled for YARN [1].
[1] https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html Best, Zhanghao Chen ________________________________ From: Vincent Woo <vincent6...@gmail.com> Sent: Monday, March 17, 2025 10:00 To: Zhanghao Chen <zhanghao.c...@outlook.com> Cc: u...@flink.apache.org <u...@flink.apache.org>; user-zh@flink.apache.org <user-zh@flink.apache.org> Subject: Re: Hardware CPU vCore fetch issue when Flink is running on yarn Thanks for the solution suggestion, I verified using Flink 1.20 + JDK 17 and it seems to still have a similar problem, and strangely enough there is also randomness in different SQL jobs in the same environment (e.g. Flink 1.17 + Hadoop 2.6 + jdk8u192) and they have the same dependency jar. [粘贴的图形-1.png] 2025年3月16日 22:10，Zhanghao Chen <zhanghao.c...@outlook.com> 写道： Hi, Vincent. Old versions of JDK8 lack proper container awareness. It is suggested to upgrade your JDK to at least 8u372 or 11.0.16, see [1][2] for more details. [1] https://bugs.openjdk.org/browse/JDK-8146115 [2] https://bugs.openjdk.org/browse/JDK-8230305 Best, Zhanghao Chen ________________________________ From: Vincent Woo <vincent6...@gmail.com> Sent: Sunday, March 16, 2025 12:09 To: u...@flink.apache.org <u...@flink.apache.org>; user-zh@flink.apache.org <user-zh@flink.apache.org> Subject: Hardware CPU vCore fetch issue when Flink is running on yarn Hello everyone, I've got a question for you all. When Flink attempts to obtain the current number of available CPU cores through the Java runtime (Runtime.getRuntime().availableProcessors()), an issue arises. Specifically, when running multiple Flink jobs, either version 1.13 or 1.17, on Yarn, we've noticed that some jobs retrieve the total number of cores of the entire machine, while others only get the cores allocated to the Yarn container. This discrepancy occurs regardless of whether it's Flink 1.13 or 1.17. Could anyone tell me what might be causing this? Here's the environmental information: Flink versions: Flink 1.13 and Flink 1.17 Hadoop: Version 2.6.0 Java: Version 1.8 Thank you! Best regards, Vincent Woo
