[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
**
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
Differences in the controls comparing to cgroup v1:
h3. *Hard limit on memory*
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. *Soft limit on memory*
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. *_Swappiness_*
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> **
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
**
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
Differences in the controls comparing to cgroup v1:
h3. *Hard limit on memory*
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. *Soft limit on memory*
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. *_Swappiness_*
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about checking if
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46]
need any updates.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> Differences in the controls comparing to cgroup v1:
> h3. *Hard limit on memory*
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. *Soft limit on memory*
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. *_Swappiness_*
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11675: -- Labels: pull-request-available (was: ) > Update MemoryResourceHandler implementation for cgroup v2 support > - > > Key: YARN-11675 > URL: https://issues.apache.org/jira/browse/YARN-11675 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > MemoryResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
