[ https://issues.apache.org/jira/browse/YARN-11669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Teke updated YARN-11669: --------------------------------- Description: cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 already moved to cgroup v2 as a default, hence YARN should support it. This umbrella tracks the required work. [Documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html] A way to test the newly added features: # Turn on cgroup v1 based on the current [documentation|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html]. # System prerequisites: ## the file {{/etc/mtab}} should contain a mount path with the file system type {{cgroup2}}, by default this could be {{/sys/fs/cgroup}} on most OS's ## the {{cgroup.subtree_control}} file should contain the necessary controllers (update it with: {{echo "+cpu +io +memory" > cgroup.subtree_control}}) ## either create the YARN hierarchy and make it owned by the user running the NM on the node. The hierarchy is {{hadoop-yarn}} by default (controller by {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}), and {{chown -R user:group /sys/fs/cgroup/hadoop-yarn}} is needed. -R is required, because as soon as the directory is created it will be filled with the controller files which YARN will try to edit. ### Alternatively if the NM process user has access rights on the {{/sys/fs/cgroup}} directory it'll try to create the hierarchy and update the {{cgroup.subtree_control}} file. # YARN configuration ## {{yarn.nodemanager.linux-container-executor.cgroups.mount-path}} should point to the directory where the cgroup2 structure is mounted and the {{hadoop-yarn}} hierarchy was created ## {{yarn.nodemanager.linux-container-executor.cgroups.v2.enabled}} should be set to {{true}} ## Enable a cgroup controller, like {{yarn. nodemanager. resource. cpu.enabled}}: {{true}} # Launch the NM and monitor the cgroup files on container launches (i.e: {{/sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight}}) was: cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 already moved to cgroup v2 as a default, hence YARN should support it. This umbrella tracks the required work. [Documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html] A way to test the newly added features: # Turn on cgroup v1 based on the current [documentation|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html]. # System prerequisites: ## the file {{/etc/mtab}} should contain a mount path with the file system type {{cgroup2}}, by default this could be {{/sys/fs/cgroup}} on most OS's ## the {{cgroup.subtree_control}} file should contain the necessary controllers (update it with: {{echo "+cpu +io +memory" > cgroup.subtree_control}}) ## either create the YARN hierarchy and make it owned by the user running the NM on the node. The hierarchy is {{hadoop-yarn}} by default (controller by {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}), and {{chown -R yarn:hadoop /sys/fs/cgroup/hadoop-yarn}} is needed. -R is required, because as soon as the directory is created it will be filled with the controller files which YARN will try to edit. ### Alternatively if the NM process user has access rights on the {{/sys/fs/cgroup}} directory it'll try to create the hierarchy and update the {{cgroup.subtree_control}} file. # YARN configuration ## {{yarn.nodemanager.linux-container-executor.cgroups.mount-path}} should point to the directory where the cgroup2 structure is mounted and the {{hadoop-yarn}} hierarchy was created ## {{yarn.nodemanager.linux-container-executor.cgroups.v2.enabled}} should be set to {{true}} ## Enable a cgroup controller, like {{yarn. nodemanager. resource. cpu.enabled}}: {{true}} # Launch the NM and monitor the cgroup files on container launches (i.e: {{/sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight}}) > [Umbrella] cgroup v2 support > ---------------------------- > > Key: YARN-11669 > URL: https://issues.apache.org/jira/browse/YARN-11669 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn > Reporter: Ferenc Erdelyi > Assignee: Benjamin Teke > Priority: Major > > cgroup v2 has some fundamental changes compared to v1. RHEL9, Ubuntu 22 > already moved to cgroup v2 as a default, hence YARN should support it. This > umbrella tracks the required work. > [Documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html] > A way to test the newly added features: > # Turn on cgroup v1 based on the current > [documentation|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html]. > # System prerequisites: > ## the file {{/etc/mtab}} should contain a mount path with the file system > type {{cgroup2}}, by default this could be {{/sys/fs/cgroup}} on most OS's > ## the {{cgroup.subtree_control}} file should contain the necessary > controllers (update it with: {{echo "+cpu +io +memory" > > cgroup.subtree_control}}) > ## either create the YARN hierarchy and make it owned by the user running the > NM on the node. The hierarchy is {{hadoop-yarn}} by default (controller by > {{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}), and {{chown > -R user:group /sys/fs/cgroup/hadoop-yarn}} is needed. -R is required, > because as soon as the directory is created it will be filled with the > controller files which YARN will try to edit. > ### Alternatively if the NM process user has access rights on the > {{/sys/fs/cgroup}} directory it'll try to create the hierarchy and update the > {{cgroup.subtree_control}} file. > # YARN configuration > ## {{yarn.nodemanager.linux-container-executor.cgroups.mount-path}} should > point to the directory where the cgroup2 structure is mounted and the > {{hadoop-yarn}} hierarchy was created > ## {{yarn.nodemanager.linux-container-executor.cgroups.v2.enabled}} should be > set to {{true}} > ## Enable a cgroup controller, like {{yarn. nodemanager. resource. > cpu.enabled}}: {{true}} > # Launch the NM and monitor the cgroup files on container launches (i.e: > {{/sys/fs/cgroup/hadoop-yarn/container_id/cpu.weight}}) -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org