[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841525#comment-17841525
]
ASF GitHub Bot commented on YARN-11191:
---
tomicooler commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081123115
@slfan1989 Yes, sure. Re-triggered.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841465#comment-17841465
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2080551000
@tomicooler Can we retrigger compilation?
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841464#comment-17841464
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on PR #6769:
URL: https://github.com/apache/hadoop/pull/6769#issuecomment-2080548182
@tomicooler Thanks for the contribution!
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841463#comment-17841463
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 merged PR #6769:
URL: https://github.com/apache/hadoop/pull/6769
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841296#comment-17841296
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079729101
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 105m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 44c51a75455e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 18421d720446fa982e620bd77df0b4f3420d533b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/testReport/ |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841286#comment-17841286
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2079693838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 2s | | trunk passed |
| +1 :green_heart: | compile | 7m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 4s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 54s | | the patch passed |
| +1 :green_heart: | compile | 7m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 7m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 7m 15s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 55s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 19 new + 174 unchanged
- 0 fixed = 193 total (was 174) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 31s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 14s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 55s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 58s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 232m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 1251bfb621b7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/had
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841274#comment-17841274
]
ASF GitHub Bot commented on YARN-11675:
---
p-szucs commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079619303
Thanks @brumi1024 for the review! Rebased and updated the PR with the fix
for your comment.
Added a description on the jira ticket as well summarizing the changes
comparing to cgroupv1: https://issues.apache.org/jira/browse/YARN-11675
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. Swappiness
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
**
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
Differences in the controls comparing to cgroup v1:
h3. *Hard limit on memory*
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. *Soft limit on memory*
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. *_Swappiness_*
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> **
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory co
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
h3. *Differences in the controls comparing to cgroup v1:*
**
h3. Hard limit on memory
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. Soft limit on memory
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. Swappiness
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory contr
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Szucs updated YARN-11675:
---
Description:
cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about updating
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
Differences in the controls comparing to cgroup v1:
h3. *Hard limit on memory*
{_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
h3. *Soft limit on memory*
{_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
Detailed descriptions about the memory controls can be found in the official
[cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
h3. *_Swappiness_*
_memory.swappiness_ has been removed from the available cgroup v2 controls.
Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
{quote}Swappiness is a property for the Linux kernel that changes the balance
between swapping out runtime memory, as opposed to dropping pages from the
system page cache. Swappiness can be set to values between 0 and 100,
inclusive. A low value means the kernel will try to avoid swapping as much as
possible where a higher value instead will make the kernel aggressively try to
use swap space.
{quote}
Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
case study we found that most of the time swappiness didn't work as expected as
it mostly depends on the I/O balance of the system, so it is no longer
available in cgroup v2.
was:cgroup v2 has some changes in various controllers (some changed their
functionality, some were removed). This task is about checking if
MemoryResourceHandler's
[implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46]
need any updates.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> Differences in the controls comparing to cgroup v1:
> h3. *Hard limit on memory*
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. *Soft limit on memory*
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. *_Swappiness_*
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841260#comment-17841260
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079518586
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 39s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 113m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dccf6577322e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b476d2d6a376f80b2ceeb111a62c66e84d5319fd |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841257#comment-17841257
]
ASF GitHub Bot commented on YARN-11675:
---
brumi1024 commented on code in PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#discussion_r1581108211
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java:
##
@@ -99,81 +66,31 @@ int getSwappiness() {
}
@Override
- public List reacquireContainer(ContainerId containerId)
- throws ResourceHandlerException {
-return null;
- }
-
- @Override
- public List updateContainer(Container container)
+ protected void updateMemoryHardLimit(String cgroupId, long
containerHardLimit)
throws ResourceHandlerException {
-String cgroupId = container.getContainerId().toString();
-File cgroup = new File(cGroupsHandler.getPathForCGroup(MEMORY, cgroupId));
-if (cgroup.exists()) {
- //memory is in MB
- long containerSoftLimit =
- (long) (container.getResource().getMemorySize() * this.softLimit);
- long containerHardLimit = container.getResource().getMemorySize();
- if (enforce) {
-try {
- cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
- CGroupsHandler.CGROUP_PARAM_MEMORY_HARD_LIMIT_BYTES,
- String.valueOf(containerHardLimit) + "M");
- ContainerTokenIdentifier id =
container.getContainerTokenIdentifier();
- if (id != null && id.getExecutionType() ==
- ExecutionType.OPPORTUNISTIC) {
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
-String.valueOf(OPPORTUNISTIC_SOFT_LIMIT) + "M");
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SWAPPINESS,
-String.valueOf(OPPORTUNISTIC_SWAPPINESS));
- } else {
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
-String.valueOf(containerSoftLimit) + "M");
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SWAPPINESS,
-String.valueOf(swappiness));
- }
-} catch (ResourceHandlerException re) {
- cGroupsHandler.deleteCGroup(MEMORY, cgroupId);
- LOG.warn("Could not update cgroup for container", re);
- throw re;
-}
- }
-}
-return null;
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
+CGroupsHandler.CGROUP_PARAM_MEMORY_HARD_LIMIT_BYTES,
+String.valueOf(containerHardLimit) + "M");
}
@Override
- public List preStart(Container container)
- throws ResourceHandlerException {
-String cgroupId = container.getContainerId().toString();
-cGroupsHandler.createCGroup(MEMORY, cgroupId);
-updateContainer(container);
-List ret = new ArrayList<>();
-ret.add(new PrivilegedOperation(
-PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP,
-PrivilegedOperation.CGROUP_ARG_PREFIX
-+ cGroupsHandler.getPathForCGroupTasks(MEMORY, cgroupId)));
-return ret;
+ protected void updateOpportunisticMemoryLimits(String cgroupId) throws
ResourceHandlerException {
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
+CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
+String.valueOf(OPPORTUNISTIC_SOFT_LIMIT) + "M");
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
Review Comment:
Can you please update the description of the jira and maybe explain in short
a comment what the [swappiness](https://access.redhat.com/solutions/103833)
controls, and that it's no longer available in cgroup v2 as most of the times
it didn't behave as expected?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2MemoryResourceHandlerImpl.java:
##
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in wr
[jira] [Commented] (YARN-11643) Skip unnecessary pre-check in Multi Node Placement
[
https://issues.apache.org/jira/browse/YARN-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841253#comment-17841253
]
ASF GitHub Bot commented on YARN-11643:
---
hadoop-yetus commented on PR #6426:
URL: https://github.com/apache/hadoop/pull/6426#issuecomment-2079445379
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 43s | | trunk passed |
| +1 :green_heart: | compile | 7m 28s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 06s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 179m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 52s | | the patch passed |
| +1 :green_heart: | javac | 3m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 04s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 190m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 37s | | The patch does not
generate ASF License warnings. |
| | | 510m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 80fcb844f29c 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 4c340e66690d361073ab7a746dbe4a9e7c446f2e |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6426/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6426/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Skip unnecessary pre-check in Multi Node Placement
> --
>
> Key: YARN-11643
> URL: https://issues.apache.org/jira/browse/YARN-11643
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Xie YiFan
>Assignee: Xie YiFan
>Priority: Minor
> Labels: pull-request-available
>
> When Multi Node Placement enabled, RegularContainerAllocator do a while loop
> to find one node from candidate set to allocate for a given scheduler key.
> Before do allocate, pre-check be called to check if current node satisfies
> check. If this node does not pass all checks, just continue to next node.
> {code:java}
> if (reservedContainer == null) {
> result = preCheckForNodeCandidateSet(node,
> schedulingMode, resourceLimits, schedulerKey);
> if (null != result) {
> continue;
> }
> } {code}
> But some checks are related to scheduler Key or Application which return
> PRIORITY_SKIPPED or APP_SKIPPED. It means that if first node does not pass
> check, the
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841207#comment-17841207 ] ASF GitHub Bot commented on YARN-11674: --- brumi1024 commented on PR #6751: URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2079350982 Thanks @tomicooler for the review, merged to trunk. > Update CpuResourceHandler implementation for cgroup v2 support > -- > > Key: YARN-11674 > URL: https://issues.apache.org/jira/browse/YARN-11674 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > CpuResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841206#comment-17841206 ] ASF GitHub Bot commented on YARN-11674: --- brumi1024 merged PR #6751: URL: https://github.com/apache/hadoop/pull/6751 > Update CpuResourceHandler implementation for cgroup v2 support > -- > > Key: YARN-11674 > URL: https://issues.apache.org/jira/browse/YARN-11674 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > CpuResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841168#comment-17841168
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 merged PR #6771:
URL: https://github.com/apache/hadoop/pull/6771
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841167#comment-17841167
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079194163
Thanks @tomicooler for the review, merging to trunk.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841158#comment-17841158
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079174965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | cc | 1m 14s | | the patch passed |
| +1 :green_heart: | golang | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 46s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 162m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 26c72a68eecc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed587f807d2eb5e93978a0faf1c77b4e506c9f21 |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file,
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841110#comment-17841110
]
ASF GitHub Bot commented on YARN-11674:
---
tomicooler commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580710076
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsCpuResourceHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.security.ContainerTokenIdentifier;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
+import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils;
+import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public abstract class AbstractCGroupsCpuResourceHandler implements
CpuResourceHandler {
+
+ static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsCpuResourceHandler.class);
+
+ protected CGroupsHandler cGroupsHandler;
+ private boolean strictResourceUsageMode = false;
+ private float yarnProcessors;
+ private int nodeVCores;
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int MAX_QUOTA_US = 1000 * 1000;
+ @VisibleForTesting
+ static final int MIN_PERIOD_US = 1000;
+
+ AbstractCGroupsCpuResourceHandler(CGroupsHandler cGroupsHandler) {
+this.cGroupsHandler = cGroupsHandler;
+ }
+
+ @Override
+ public List bootstrap(Configuration conf)
+ throws ResourceHandlerException {
+return bootstrap(
+ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
conf);
+ }
+
+ @VisibleForTesting
+ List bootstrap(
+ ResourceCalculatorPlugin plugin, Configuration conf)
+ throws ResourceHandlerException {
+this.strictResourceUsageMode = conf.getBoolean(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
+
YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
+this.cGroupsHandler.initializeCGroupController(CPU);
+nodeVCores = NodeManagerHardwareUtils.getVCores(plugin, conf);
+
+// cap overall usage to the number of cores allocated to YARN
+yarnProcessors = NodeManagerHardwareUtils.getContainersCPUs(plugin, conf);
+int systemProcessors = NodeManagerHardwareUtils.getNodeCPUs(plugin, conf);
+boolean existingCpuLimits;
+existingCpuLimits = cpuLimitExists(
+cGroupsHandler.getPathForCGroup(CPU, ""));
+
+if (systemProcessors != (int) yarnProcessors) {
+ LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
+ int[] limits = getOverallLimits(yarnProcessors);
+ updateCgroupMaxCpuLimit("", String.valueOf(limits[1]),
String.valueOf(limits[0]));
+} else if (existingCpuLimits) {
+ LOG.info("Removing CPU constraints for YARN containers.");
+ updateCgroupMaxCpuLimit("", String.valueOf(-1), null);
+}
+return null;
+ }
+
+ protected abstract void updateCgroupMaxCpuLimit(String cgroupId, String
quota, String period)
+
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841104#comment-17841104
]
ASF GitHub Bot commented on YARN-11674:
---
brumi1024 commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580689988
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsCpuResourceHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.security.ContainerTokenIdentifier;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
+import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils;
+import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public abstract class AbstractCGroupsCpuResourceHandler implements
CpuResourceHandler {
+
+ static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsCpuResourceHandler.class);
+
+ protected CGroupsHandler cGroupsHandler;
+ private boolean strictResourceUsageMode = false;
+ private float yarnProcessors;
+ private int nodeVCores;
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int MAX_QUOTA_US = 1000 * 1000;
+ @VisibleForTesting
+ static final int MIN_PERIOD_US = 1000;
+
+ AbstractCGroupsCpuResourceHandler(CGroupsHandler cGroupsHandler) {
+this.cGroupsHandler = cGroupsHandler;
+ }
+
+ @Override
+ public List bootstrap(Configuration conf)
+ throws ResourceHandlerException {
+return bootstrap(
+ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
conf);
+ }
+
+ @VisibleForTesting
+ List bootstrap(
+ ResourceCalculatorPlugin plugin, Configuration conf)
+ throws ResourceHandlerException {
+this.strictResourceUsageMode = conf.getBoolean(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
+
YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
+this.cGroupsHandler.initializeCGroupController(CPU);
+nodeVCores = NodeManagerHardwareUtils.getVCores(plugin, conf);
+
+// cap overall usage to the number of cores allocated to YARN
+yarnProcessors = NodeManagerHardwareUtils.getContainersCPUs(plugin, conf);
+int systemProcessors = NodeManagerHardwareUtils.getNodeCPUs(plugin, conf);
+boolean existingCpuLimits;
+existingCpuLimits = cpuLimitExists(
+cGroupsHandler.getPathForCGroup(CPU, ""));
+
+if (systemProcessors != (int) yarnProcessors) {
+ LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
+ int[] limits = getOverallLimits(yarnProcessors);
+ updateCgroupMaxCpuLimit("", String.valueOf(limits[1]),
String.valueOf(limits[0]));
+} else if (existingCpuLimits) {
+ LOG.info("Removing CPU constraints for YARN containers.");
+ updateCgroupMaxCpuLimit("", String.valueOf(-1), null);
+}
+return null;
+ }
+
+ protected abstract void updateCgroupMaxCpuLimit(String cgroupId, String
quota, String period)
+
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841102#comment-17841102
]
ASF GitHub Bot commented on YARN-11674:
---
brumi1024 commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580684787
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2CpuResourceHandlerImpl.java:
##
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+
+/**
+ * An implementation for using CGroups V2 to restrict CPU usage on Linux. The
+ * implementation supports 3 different controls - restrict usage of all YARN
+ * containers, restrict relative usage of individual YARN containers and
+ * restrict usage of individual YARN containers. Admins can set the overall CPU
+ * to be used by all YARN containers - this is implemented by setting
+ * cpu.max to the value desired. If strict resource usage mode is not enabled,
+ * cpu.weight is set for individual containers - this prevents containers from
+ * exceeding the overall limit for YARN containers but individual containers
+ * can use as much of the CPU as available(under the YARN limit). If strict
+ * resource usage is enabled, then container can only use the percentage of
+ * CPU allocated to them and this is again implemented using cpu.max.
+ */
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public class CGroupsV2CpuResourceHandlerImpl extends
AbstractCGroupsCpuResourceHandler {
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int CPU_DEFAULT_WEIGHT = 100; // cgroup v2 default
+ static final int CPU_DEFAULT_WEIGHT_OPPORTUNISTIC = 1;
+ static final int CPU_MAX_WEIGHT = 1;
+ static final String NO_LIMIT = "max";
+
+
+ CGroupsV2CpuResourceHandlerImpl(CGroupsHandler cGroupsHandler) {
+super(cGroupsHandler);
+ }
+
+ @Override
+ protected void updateCgroupMaxCpuLimit(String cgroupId, String max, String
period)
+ throws ResourceHandlerException {
+String cpuMaxLimit = cGroupsHandler.getCGroupParam(CPU, cgroupId,
Review Comment:
Done.
> Update CpuResourceHandler implementation for cgroup v2 support
> --
>
> Key: YARN-11674
> URL: https://issues.apache.org/jira/browse/YARN-11674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about checking if
> CpuResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60]
> need any updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11323) [Federation] Improve Router Handler FinishApps
[ https://issues.apache.org/jira/browse/YARN-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841101#comment-17841101 ] ASF GitHub Bot commented on YARN-11323: --- liubin101 commented on PR #4954: URL: https://github.com/apache/hadoop/pull/4954#issuecomment-2078901927 @slfan1989 Hi, I found that newly submitted apps may be mistakenly deleted in Delete triggered condition 1: If an app was submitted to Router, the Router stores this app in state store, but it hasn't been submitted to SubCluster. At this time, the CleanUp thread compares the lists queried in the state store and the SubCluster. This app exists in the state store but not in the SubCluster, so it will be mistakenly deleted. > [Federation] Improve Router Handler FinishApps > -- > > Key: YARN-11323 > URL: https://issues.apache.org/jira/browse/YARN-11323 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router, yarn >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841100#comment-17841100
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580673804
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
Done.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11644) LogAggregationService can't upload log in time when application finished
[
https://issues.apache.org/jira/browse/YARN-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841055#comment-17841055
]
ASF GitHub Bot commented on YARN-11644:
---
hadoop-yetus commented on PR #6452:
URL: https://github.com/apache/hadoop/pull/6452#issuecomment-2078653415
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 32s | | trunk passed |
| +1 :green_heart: | compile | 5m 11s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 149m 06s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 01s | | the patch passed |
| +1 :green_heart: | compile | 2m 39s | | the patch passed |
| +1 :green_heart: | javac | 2m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 01s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 162m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 35s | | The patch does not
generate ASF License warnings. |
| | | 426m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6452 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 8b443c886006 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 19826d01ffafb3bd2d371ef27f8efe86739f666d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6452/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6452/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> LogAggregationService can't upload log in time when application finished
>
>
> Key: YARN-11644
> URL: https://issues.apache.org/jira/browse/YARN-11644
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Affects Versions: 3.3.6
>Reporter: Xie YiFan
>Assignee: Xie YiFan
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-01-10-11-03-57-553.png
>
>
> LogAggregationService is responsible for uploading log to HDFS. It applies
> thread pool to execute upload task.
> The workflow of upload log as follow:
> # NM construct Applicaiton object when first container of a certain
> application launch, then notify LogAggregationService to init
> AppLogAggregationImpl.
> # LogAggregationService submit AppLogAggregationImpl to task queue
> # The idle worker of thread pool pulls AppLogAggregationImpl from task queue.
> # AppLogAggregationImpl do while loop to check the application state, do
> upload when application fin
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[
https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841015#comment-17841015
]
ASF GitHub Bot commented on YARN-11649:
---
hadoop-yetus commented on PR #6481:
URL: https://github.com/apache/hadoop/pull/6481#issuecomment-2078481513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 90m 10s | | trunk passed |
| +1 :green_heart: | compile | 11m 27s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 15m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 15m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 171m 00s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 41s | | the patch passed |
| +1 :green_heart: | javac | 9m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 10m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 10m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 178m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 45s | | The patch does not
generate ASF License warnings. |
| | | 488m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6481 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 e18c298462c5 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 1349aec7840f391e4229fd7a1b0d25159d9aa74d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6481/1/testReport/
|
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6481/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> YARN Federation getNewApplication returns different maxresourcecapability
> -
>
> Key: YARN-11649
> URL: https://issues.apache.org/jira/browse/YARN-11649
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jeffrey Chang
>Assignee: Jeffrey Chang
>Priority: Minor
> Labels: pull-request-available
>
> When getNewApplication is called against YARN Router with Federation on, its
> possible we get different maxResourceCapabilities in its response on
> different calls. This is because getNewApplication is called against a random
> cluster on each call, which may return different maxResourceCapability based
> on the cluster that the call is executed on. ie in response in HTTP
> [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API]
>
--
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841014#comment-17841014
]
ASF GitHub Bot commented on YARN-11690:
---
tomicooler commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580335818
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
This must be done after the magic.h include. Let's move this to the .c file
somewhere after the include.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11654) [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
[
https://issues.apache.org/jira/browse/YARN-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840902#comment-17840902
]
ASF GitHub Bot commented on YARN-11654:
---
hadoop-yetus commented on PR #6528:
URL: https://github.com/apache/hadoop/pull/6528#issuecomment-2077874147
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 94m 09s | | trunk passed |
| +1 :green_heart: | compile | 5m 02s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 151m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 09s | | the patch passed |
| +1 :green_heart: | compile | 2m 36s | | the patch passed |
| +1 :green_heart: | javac | 2m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 08s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 157m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 44s | | The patch does not
generate ASF License warnings. |
| | | 427m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6528 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9d60b5d6be65 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / be835e44742a7755c26d1b26d196db8cb2894e6a |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6528/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6528/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
>
>
> Key: YARN-11654
> URL: https://issues.apache.org/jira/browse/YARN-11654
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Labels: pull-request-available
>
> [ERROR] TestLinuxContainerExecutorWithMocks.testStartLocalizer:310
> Expected size:<26> but was:<28> in:
> <["nobody",
> "test",
> "0",
> "application_0",
> "12345",
> "/bin/nmPrivateCTokensPath",
>
> "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/tmp/nm-local-dir",
> "src/test/resources",
>
> "/opt/homebrew/Cellar/openjdk@17/17.0.8/libexec/openjdk.jdk/Contents/Home/bin/java",
> "-classpath",
>
> "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes:/Users/bilwa/code/hadoop/hadoop-yarn-pr
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840898#comment-17840898
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2077848714
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 6s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 27s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 6 new + 16 unchanged - 0 fixed = 22 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 51s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 158m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d278dec01c5c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / af079ef449f6cb73b58c822531b834ac5fbfd125 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/7/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840895#comment-17840895
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2077839495
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 54s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 26s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 6 new + 16 unchanged - 0 fixed = 22 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 157m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c528f83c64ab 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / af079ef449f6cb73b58c822531b834ac5fbfd125 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/6/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840867#comment-17840867
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2077695501
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 45m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 51m 7s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed |
| +1 :green_heart: | cc | 1m 13s | | the patch passed |
| +1 :green_heart: | golang | 1m 13s | | the patch passed |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 48s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 206m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 3dd51f4f1063 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c9e812da5d2ca575563caa8c0b59351419ebb44 |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file,
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840855#comment-17840855
]
ASF GitHub Bot commented on YARN-11684:
---
slfan1989 merged PR #6753:
URL: https://github.com/apache/hadoop/pull/6753
> PriorityQueueComparator violates general contract
> -
>
> Key: YARN-11684
> URL: https://issues.apache.org/jira/browse/YARN-11684
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.5.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1, 3.5.0
>
>
> YARN-10178 tried to fix the issue but there are still 2 property that might
> change during sorting which causes an exception.
> {code}
> 2024-04-10 12:36:56,420 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[Thread-28,5,main] threw an Exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeCollapse(TimSort.java:441)
> at java.util.TimSort.sort(TimSort.java:245)
> at java.util.Arrays.sort(Arrays.java:1512)
> at
> java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348)
> at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:260)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:1100)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:942)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1719)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1654)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1811)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1557)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:539)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:591)
> {code}
> The `queue.getAccessibleNodeLabels()` and `queue.getPriority()` could change
> in another thread while the `queues` are being sorted. Those should be saved
> when constructing the PriorityQueueResourcesForSorting helper object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840856#comment-17840856
]
ASF GitHub Bot commented on YARN-11684:
---
slfan1989 commented on PR #6753:
URL: https://github.com/apache/hadoop/pull/6753#issuecomment-2077624938
@tomicooler Thanks for the contribution! Merged into branch-3.4.
> PriorityQueueComparator violates general contract
> -
>
> Key: YARN-11684
> URL: https://issues.apache.org/jira/browse/YARN-11684
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.5.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1, 3.5.0
>
>
> YARN-10178 tried to fix the issue but there are still 2 property that might
> change during sorting which causes an exception.
> {code}
> 2024-04-10 12:36:56,420 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[Thread-28,5,main] threw an Exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeCollapse(TimSort.java:441)
> at java.util.TimSort.sort(TimSort.java:245)
> at java.util.Arrays.sort(Arrays.java:1512)
> at
> java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348)
> at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:260)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:1100)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:942)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1719)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1654)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1811)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1557)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:539)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:591)
> {code}
> The `queue.getAccessibleNodeLabels()` and `queue.getPriority()` could change
> in another thread while the `queues` are being sorted. Those should be saved
> when constructing the PriorityQueueResourcesForSorting helper object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840843#comment-17840843
]
ASF GitHub Bot commented on YARN-11674:
---
tomicooler commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1579625169
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2CpuResourceHandlerImpl.java:
##
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+
+/**
+ * An implementation for using CGroups V2 to restrict CPU usage on Linux. The
+ * implementation supports 3 different controls - restrict usage of all YARN
+ * containers, restrict relative usage of individual YARN containers and
+ * restrict usage of individual YARN containers. Admins can set the overall CPU
+ * to be used by all YARN containers - this is implemented by setting
+ * cpu.max to the value desired. If strict resource usage mode is not enabled,
+ * cpu.weight is set for individual containers - this prevents containers from
+ * exceeding the overall limit for YARN containers but individual containers
+ * can use as much of the CPU as available(under the YARN limit). If strict
+ * resource usage is enabled, then container can only use the percentage of
+ * CPU allocated to them and this is again implemented using cpu.max.
+ */
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public class CGroupsV2CpuResourceHandlerImpl extends
AbstractCGroupsCpuResourceHandler {
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int CPU_DEFAULT_WEIGHT = 100; // cgroup v2 default
+ static final int CPU_DEFAULT_WEIGHT_OPPORTUNISTIC = 1;
+ static final int CPU_MAX_WEIGHT = 1;
+ static final String NO_LIMIT = "max";
+
+
+ CGroupsV2CpuResourceHandlerImpl(CGroupsHandler cGroupsHandler) {
+super(cGroupsHandler);
+ }
+
+ @Override
+ protected void updateCgroupMaxCpuLimit(String cgroupId, String max, String
period)
+ throws ResourceHandlerException {
+String cpuMaxLimit = cGroupsHandler.getCGroupParam(CPU, cgroupId,
Review Comment:
Maybe a small doc here about the file format:
https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files
```
A read-write two value file which exists on non-root cgroups. The default is
“max 10”.
The maximum bandwidth limit. It’s in the following format:
$MAX $PERIOD
which indicates that the group may consume up to $MAX in each $PERIOD
duration. “max” for $MAX indicates no limit. If only one number is written,
$MAX is updated.
```
rename: cpuMaxLimit -> currentCpuMax
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsCpuResourceHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language govern
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840819#comment-17840819
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2077302452
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 90m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | cc | 1m 14s | | the patch passed |
| +1 :green_heart: | golang | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 29s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 161m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 32f47319a7b7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 64256e8f1b3ce6fb9f9bf79c8b9c04a057f7b72c |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/testReport/ |
| Max. process+thread count | 599 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file,
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840764#comment-17840764
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1579314396
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:
##
@@ -242,8 +242,8 @@ static int write_pid_to_cgroup_as_root(const char*
cgroup_file, pid_t pid) {
strerror(errno));
rc = -1;
goto cleanup;
- } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
-fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
cgroup_file);
+ } else if (buf.f_type != CGROUP_SUPER_MAGIC || buf.f_type !=
CGROUP2_SUPER_MAGIC) {
Review Comment:
Thanks, this should be the reason. Added.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840765#comment-17840765
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2076967275
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 25s | | trunk passed |
| +1 :green_heart: | compile | 7m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 6m 58s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 58s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 47s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 48s | | the patch passed |
| +1 :green_heart: | compile | 6m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 6m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 3s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 7m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 50s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 174 unchanged
- 0 fixed = 194 total (was 174) |
| +1 :green_heart: | mvnsite | 2m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 55s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 219m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 09e01c5a5895 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2cc0ba7c392adc4104b02de9e8efa87868ca8011 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840750#comment-17840750
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2076880252
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 59s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 158m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d3ea686fcb20 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 189d2b15576e176bb50ddd0f4577a3d1dd89c2fd |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/5/testReport/ |
| Max. process+thread count
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840749#comment-17840749
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2076877632
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 55s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 159m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 956133ce6ff5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 189d2b15576e176bb50ddd0f4577a3d1dd89c2fd |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/4/testReport/ |
| Max. process+thread count
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840745#comment-17840745
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2076847309
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 50s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 32s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 31s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 36s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 37s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 20m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 87m 5s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 172m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6768 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 111d4e1feebc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 985da47eda7acc4d92a0a5c4095c164f33520725 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/2/testReport/ |
| Max. process+thread count | 955 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/2/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840725#comment-17840725
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2076761316
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 19 new + 9 unchanged - 0 fixed = 28 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 43s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 21s | | The patch does not
generate ASF License warnings. |
| | | 107m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 79073a6720ec 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 66c0fff3b6c00028d58a2f7188db916de5c35d5e |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/3/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840705#comment-17840705
]
ASF GitHub Bot commented on YARN-11690:
---
tomicooler commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1579096249
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:
##
@@ -242,8 +242,8 @@ static int write_pid_to_cgroup_as_root(const char*
cgroup_file, pid_t pid) {
strerror(errno));
rc = -1;
goto cleanup;
- } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
-fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
cgroup_file);
+ } else if (buf.f_type != CGROUP_SUPER_MAGIC || buf.f_type !=
CGROUP2_SUPER_MAGIC) {
Review Comment:
https://docs.kernel.org/admin-guide/cgroup-v2.html
It should come from
https://github.com/torvalds/linux/blob/v4.6/include/uapi/linux/magic.h#L58
which we include, probably it is an old kernel.
Could be fixed with:
```
#ifndef CGROUP2_SUPER_MAGIC
#define CGROUP2_SUPER_MAGIC 0x63677270
#endif
```
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11655) modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler webui from -1 to 0
[
https://issues.apache.org/jira/browse/YARN-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840645#comment-17840645
]
ASF GitHub Bot commented on YARN-11655:
---
hadoop-yetus commented on PR #6564:
URL: https://github.com/apache/hadoop/pull/6564#issuecomment-2076330111
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 90m 42s | | trunk passed |
| +1 :green_heart: | compile | 5m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 02s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 144m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 04s | | the patch passed |
| +1 :green_heart: | compile | 2m 26s | | the patch passed |
| +1 :green_heart: | javac | 2m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 04s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 157m 00s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 26s | | The patch does not
generate ASF License warnings. |
| | | 416m 04s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6564 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 3ef5ad7297f5 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 6dd3f57c6e90f0aeb92a042e7306db9345cdd524 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6564/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6564/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler
> webui from -1 to 0
> --
>
> Key: YARN-11655
> URL: https://issues.apache.org/jira/browse/YARN-11655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-common
>Affects Versions: 3.3.3
>Reporter: wangzhongwei
>Assignee: wangzhongwei
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-02-20-15-15-34-996.png,
> image-2024-02-20-15-24-09-524.png
>
>
> in yarn scheduler webui,the value of Allocated GPUs and Reserved GPUs be set
> to 0 by default may be better. when GPUs not used,these values should be 0
> !image-2024-02-20-15-15-34-996.png|width=486,height=235!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840637#comment-17840637
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-2076287327
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 05s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 89m 23s | | trunk passed |
| +1 :green_heart: | compile | 11m 21s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 05s | | trunk passed |
| +1 :green_heart: | mvnsite | 11m 06s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 159m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 3m 06s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 6m 16s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| -1 :x: | javac | 6m 16s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 03s | | the patch passed |
| -1 :x: | mvnsite | 3m 26s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | javadoc | 7m 30s | | the patch passed |
| -1 :x: | shadedclient | 85m 23s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 4m 25s | | The patch does not
generate ASF License warnings. |
| | | 375m 07s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6569 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9bb75f535c07 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 465a0de4f481ee2dc7383e6fb40b93e32135ba3c |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/testReport/
|
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6569/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RMStateStore event queue blocked
>
>
> Key: YARN-11656
> URL: https://issues.
[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840605#comment-17840605
]
ASF GitHub Bot commented on YARN-11664:
---
hadoop-yetus commented on PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#issuecomment-2076109814
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 01s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 85m 46s | | trunk passed |
| +1 :green_heart: | compile | 37m 36s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 39s | | trunk passed |
| -1 :x: | mvnsite | 4m 28s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6631/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 23m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 186m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 12s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 17m 49s | | the patch passed |
| +1 :green_heart: | compile | 34m 55s | | the patch passed |
| +1 :green_heart: | javac | 34m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 38s | | the patch passed |
| -1 :x: | mvnsite | 4m 14s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6631/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 23m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 191m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 21s | | The patch does not
generate ASF License warnings. |
| | | 570m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6631 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 d6fe0eef25f3 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ac54ffe34f312cf944c686c788eb37f311e77827 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6631/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6631/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires
[jira] [Commented] (YARN-11261) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy
[
https://issues.apache.org/jira/browse/YARN-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840576#comment-17840576
]
ASF GitHub Bot commented on YARN-11261:
---
hadoop-yetus commented on PR #6652:
URL: https://github.com/apache/hadoop/pull/6652#issuecomment-2075830470
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 83m 53s | | trunk passed |
| +1 :green_heart: | compile | 39m 06s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 54s | | trunk passed |
| -1 :x: | mvnsite | 4m 26s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6652/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 10m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 160m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 16s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 57s | | the patch passed |
| +1 :green_heart: | compile | 36m 20s | | the patch passed |
| +1 :green_heart: | javac | 36m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 53s | | the patch passed |
| -1 :x: | mvnsite | 4m 25s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6652/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 10m 06s | | the patch passed |
| +1 :green_heart: | shadedclient | 168m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 31s | | The patch does not
generate ASF License warnings. |
| | | 514m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6652 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 8fff8b2aa0df 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 0fbb692f24d2a286c36ad771f7f57b5fe3f3c8cd |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6652/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6652/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy
> -
>
> Key: YARN-11261
> URL: https://issues.apache.org/jira/browse/YARN-11261
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Affects Versions: 3.3.4
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840570#comment-17840570
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2075790931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 44m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| -1 :x: | compile | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | cc | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | golang | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | javac | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 50s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 181m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux b41f66700d8a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a3beaf9d526f13c4cd22b9e8ee1de470a4462c5a |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840559#comment-17840559
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2075756590
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 6 new + 16 unchanged - 0 fixed = 22 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| -1 :x: | asflicense | 0m 37s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/5/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 156m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b6f30a60d4ad 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fae4115a06fb5a054305e3eec2ad1f759077ad5d |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840543#comment-17840543
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6769:
URL: https://github.com/apache/hadoop/pull/6769#issuecomment-2075632733
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 58s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 57s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 47s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 1s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 48s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 58s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 41m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 95m 54s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 254m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6769/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6769 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a214af58c8fa 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 7c732b60916ebadef8ec7b06a634aaef28a76047 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6769/1/testReport/ |
| Max. process+thread count | 857 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6769/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQue
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840526#comment-17840526
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 opened a new pull request, #6771:
URL: https://github.com/apache/hadoop/pull/6771
### Description of PR
The container executor function write_pid_to_cgroup_as_root writes the PID
of the newly launched container to the correct cgroup.procs file. However it
checks if the file is mounted on a cgroup filesystem, and does that check using
the magic number, which differs for v1 and v2. This should handle v1 or v2
filesystems as well.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11690:
--
Labels: pull-request-available (was: )
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
Benjamin Teke created YARN-11690:
Summary: Update container executor to use CGROUP2_SUPER_MAGIC in
cgroup 2 scenarios
Key: YARN-11690
URL: https://issues.apache.org/jira/browse/YARN-11690
Project: Hadoop YARN
Issue Type: Sub-task
Components: container-executor
Reporter: Benjamin Teke
Assignee: Benjamin Teke
The container executor function {{write_pid_to_cgroup_as_root}} writes the PID
of the newly launched container to the correct cgroup.procs file. However it
checks if the file is mounted on a cgroup filesystem, and does that check using
the magic number, which differs for v1 and v2. This should handle v1 or v2
filesystems as well.
{code:java}
/**
* Write the pid of the current process to the cgroup file.
* cgroup_file: Path to cgroup file where pid needs to be written to.
*/
static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
int rc = 0;
uid_t user = geteuid();
gid_t group = getegid();
if (change_effective_user(0, 0) != 0) {
rc = -1;
goto cleanup;
}
// statfs
struct statfs buf;
if (statfs(cgroup_file, ) == -1) {
fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n", cgroup_file,
strerror(errno));
rc = -1;
goto cleanup;
} else if (buf.f_type != CGROUP_SUPER_MAGIC) {
fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
cgroup_file);
rc = -1;
goto cleanup;
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840510#comment-17840510
]
ASF GitHub Bot commented on YARN-11387:
---
hadoop-yetus commented on PR #6660:
URL: https://github.com/apache/hadoop/pull/6660#issuecomment-2075366509
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 97m 37s | | trunk passed |
| +1 :green_heart: | compile | 4m 48s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 158m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 2m 08s | | the patch passed |
| +1 :green_heart: | javac | 2m 08s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 03s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 06s | | the patch passed |
| +1 :green_heart: | shadedclient | 169m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 43s | | The patch does not
generate ASF License warnings. |
| | | 448m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 21b7d79e94d7 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 729bac3f2f5d8b18ac4c422a34797b3fb3626d6b |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6660/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6660/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840493#comment-17840493
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2075284019
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 3s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 34s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 33s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 34s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 38s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 37s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 10s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 20m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 27s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 27s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| -1 :x: | mvnsite | 0m 24s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server
[jira] [Created] (YARN-11689) Update getErrorWithDetails method to provide more meaningful error messages
Benjamin Teke created YARN-11689: Summary: Update getErrorWithDetails method to provide more meaningful error messages Key: YARN-11689 URL: https://issues.apache.org/jira/browse/YARN-11689 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke The method AbstractCGroupsHandler.getErrorWithDetails hides quite a lot of information. It would be useful to show the underlying exception and it's message as well, by default. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11689) Update getErrorWithDetails method to provide more meaningful error messages
[ https://issues.apache.org/jira/browse/YARN-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-11689: Assignee: Benjamin Teke > Update getErrorWithDetails method to provide more meaningful error messages > --- > > Key: YARN-11689 > URL: https://issues.apache.org/jira/browse/YARN-11689 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > The method AbstractCGroupsHandler.getErrorWithDetails hides quite a lot of > information. It would be useful to show the underlying exception and it's > message as well, by default. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840465#comment-17840465 ] ASF GitHub Bot commented on YARN-11685: --- p-szucs opened a new pull request, #6770: URL: https://github.com/apache/hadoop/pull/6770 ### Description of PR Various OS's mount the cgroup v2 differently, some of them mount both the v1 and v2 structure, others mount a hybrid structure. To avoid initialization issues the cgroup v1/v2 functionality should be set by a config property. ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11685: -- Labels: pull-request-available (was: ) > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840457#comment-17840457
]
ASF GitHub Bot commented on YARN-11191:
---
tomicooler opened a new pull request, #6769:
URL: https://github.com/apache/hadoop/pull/6769
… (#6732)
(cherry picked from commit ecf665c6facf89d3b87b6e3cc684274b8155ca60)
Change-Id: I561bcad51af7810328c8b91cd9290d5198be0c6e
### Description of PR
Backport, there were conflicts (abstractparent/leaf queue, and queuepath
doesn't exist here yet).
Jira: [YARN-11191](https://issues.apache.org/jira/browse/YARN-11191)
Original PR: #6732
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840437#comment-17840437
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2074930513
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 43s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 6 new + 9 unchanged - 0 fixed = 15 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 43s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| -1 :x: | asflicense | 0m 38s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/4/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 156m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f15780c3500a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 40337376510cafe4b6283a2531fb098ec76ac7b3 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840432#comment-17840432
]
ASF GitHub Bot commented on YARN-11191:
---
tomicooler opened a new pull request, #6768:
URL: https://github.com/apache/hadoop/pull/6768
… (#6732)
(cherry picked from commit ecf665c6facf89d3b87b6e3cc684274b8155ca60)
### Description of PR
backport.
Jira: [YARN-11191](https://issues.apache.org/jira/browse/YARN-11191)
Original PR: #6732
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke reassigned YARN-11191:
Assignee: Tamas Domok
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840428#comment-17840428
]
ASF GitHub Bot commented on YARN-11191:
---
brumi1024 merged PR #6732:
URL: https://github.com/apache/hadoop/pull/6732
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840427#comment-17840427
]
ASF GitHub Bot commented on YARN-11191:
---
brumi1024 commented on PR #6732:
URL: https://github.com/apache/hadoop/pull/6732#issuecomment-2074890741
Thanks for the patch @tomicooler and @slfan1989 @p-szucs for the review,
merging to trunk. @tomicooler can you please check if a backport is possible to
the branches 3.3 and 3.4?
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[ https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840357#comment-17840357 ] ASF GitHub Bot commented on YARN-11672: --- brumi1024 commented on PR #6734: URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2074523067 Thanks for the review @tomicooler @slfan1989, merged to trunk. > Create a CgroupHandler implementation for cgroup v2 > --- > > Key: YARN-11672 > URL: https://issues.apache.org/jira/browse/YARN-11672 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > [CGroupsHandler's|https://github.com/apache/hadoop/blob/69b328943edf2f61c8fc139934420e3f10bf3813/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java#L36] > current implementation holds the functionality to mount and setup the YARN > specific cgroup v1 functionality. A similar v2 implementation should be > created that allows initialising the v2 structure. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840239#comment-17840239
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6732:
URL: https://github.com/apache/hadoop/pull/6732#issuecomment-2073537387
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 106m 35s | | trunk passed |
| +1 :green_heart: | compile | 6m 48s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 174m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 23s | | the patch passed |
| +1 :green_heart: | compile | 4m 06s | | the patch passed |
| +1 :green_heart: | javac | 4m 06s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 38s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 190m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 36s | | The patch does not
generate ASF License warnings. |
| | | 502m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6732 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 0cbe62357619 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 73c01ccb77ca85cd83785726b55841fe8b94e973 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6732/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6732/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So t
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840217#comment-17840217
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2073373773
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 118m 22s | | trunk passed |
| +1 :green_heart: | compile | 6m 57s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 04s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 185m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 42s | | the patch passed |
| +1 :green_heart: | javac | 3m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 08s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 07s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 203m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 7m 24s | | The patch does not
generate ASF License warnings. |
| | | 541m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 296da9c658f3 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 13ddcba8f489f8345e05cee965dcd90bf16255fb |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6734/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6734/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Create a CgroupHandler implementation for cgroup v2
> ---
>
> Key: YARN-11672
> URL: https://issues.apache.org/jira/browse/YARN-11672
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsHandler's|https://github.com/apache/hadoop/blob/69b328943edf2f61c8fc139934420e3f10bf3813/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java#L36]
> current implementation holds the functionality to mount and setup the YARN
> specific cgroup v1 functionality. A similar v2 implementation should be
> created that allows initialising the v2 structure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11662) RM Web API endpoint queue reference differs from JMX endpoint for CS
[ https://issues.apache.org/jira/browse/YARN-11662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi resolved YARN-11662. --- Resolution: Duplicate Duplicate of YARN-11538 > RM Web API endpoint queue reference differs from JMX endpoint for CS > > > Key: YARN-11662 > URL: https://issues.apache.org/jira/browse/YARN-11662 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > > When a placement is not successful (because of the lack of a placement rule > or an unsuccessful placement), the application is placed in the default queue > instead of the root.default. The parent queue won't be defined when there is > no placement rule. This causes an inconsistency between the JMX endpoint > (reporting the app. runs under the root.default) and the RM Web API endpoint > (reporting the app runs under the default queue). > Similarly, when we submit an application with an unambiguous leaf queue > specified, the RM Web API endpoint will report the queue as the leaf queue > name instead of the full queue path. However, the full queue path is the > expected value to be consistent with the JMX endpoint. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840082#comment-17840082
]
ASF GitHub Bot commented on YARN-11672:
---
brumi1024 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1576141237
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsHandler.java:
##
@@ -0,0 +1,579 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.SystemClock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractCGroupsHandler implements CGroupsHandler {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsHandler.class);
+ protected static final String MTAB_FILE = "/proc/mounts";
+
+ private final long deleteCGroupTimeout;
+ private final long deleteCGroupDelay;
+ private final Clock clock;
+
+ protected final String mtabFile;
+ protected final CGroupsMountConfig cGroupsMountConfig;
+ protected final ReadWriteLock rwLock;
+ protected Map controllerPaths;
+ protected Map> parsedMtab;
+ protected final PrivilegedOperationExecutor privilegedOperationExecutor;
+ protected final String cGroupPrefix;
+
+ /**
+ * Create cgroup handler object.
+ *
+ * @param confconfiguration
+ * @param privilegedOperationExecutor provides mechanisms to execute
+ *PrivilegedContainerOperations
+ * @param mtabmount file location
+ * @throws ResourceHandlerException if initialization failed
+ */
+ AbstractCGroupsHandler(Configuration conf, PrivilegedOperationExecutor
+ privilegedOperationExecutor, String mtab)
+ throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
+this.cGroupPrefix = conf.get(YarnConfiguration.
+NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
+.replaceAll("^/+", "").replaceAll("/+$", "");
+this.cGroupsMountConfig = new CGroupsMountConfig(conf);
+this.deleteCGroupTimeout = conf.getLong(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT) +
+conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
+YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) + 1000;
+this.deleteCGroupDelay =
+conf.getLong(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY);
+this.controllerPaths = new HashMap<>();
+this.parsedMtab = new HashMap<>();
+this.rwLock = new ReentrantReadWriteLock();
+this.privilegedOperationExecutor = privilegedOperationExecutor;
+
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840078#comment-17840078
]
ASF GitHub Bot commented on YARN-11684:
---
hadoop-yetus commented on PR #6753:
URL: https://github.com/apache/hadoop/pull/6753#issuecomment-2072124888
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 13m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 29s | | branch-3.4 passed |
| +1 :green_heart: | compile | 1m 5s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 56s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 2s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 1m 1s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 59s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 51s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 0s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 40m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 49s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 265m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6753/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6753 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 87ae7cc6e595 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / b133b485e45787b183269479cd54cef8b33965e7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6753/2/testReport/ |
| Max. process+thread count | 960 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6753/2/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840051#comment-17840051
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2071934931
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 52m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 41s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/17/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 9 unchanged - 2 fixed = 26 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 29s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 182m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 444088536d54 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 13ddcba8f489f8345e05cee965dcd90bf16255fb |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/17/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840042#comment-17840042
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6732:
URL: https://github.com/apache/hadoop/pull/6732#issuecomment-2071872714
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 90m 8s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 176m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6732/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6732 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e28442f7b76e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 73c01ccb77ca85cd83785726b55841fe8b94e973 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6732/2/testReport/ |
| Max. process+thread count | 941 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6732/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (YARN-11686) Correct traversing indexs when scheduling asynchronously using Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840016#comment-17840016
]
ASF GitHub Bot commented on YARN-11686:
---
hadoop-yetus commented on PR #6750:
URL: https://github.com/apache/hadoop/pull/6750#issuecomment-2071760148
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 08s | | trunk passed |
| +1 :green_heart: | compile | 6m 00s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 147m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 36s | | the patch passed |
| +1 :green_heart: | compile | 3m 11s | | the patch passed |
| +1 :green_heart: | javac | 3m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 02s | | the patch passed |
| +1 :green_heart: | shadedclient | 158m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 33s | | The patch does not
generate ASF License warnings. |
| | | 420m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6750 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 e0c4ad74b72d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 0e1ae3a46317d9941462ffc5f2aaf3271d0b3390 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6750/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6750/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Correct traversing indexs when scheduling asynchronously using Capacity
> Scheduler
> -
>
> Key: YARN-11686
> URL: https://issues.apache.org/jira/browse/YARN-11686
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yihe Li
>Priority: Minor
> Labels: pull-request-available
>
> When scheduling asynchronously using Capacity Scheduler, the traversing
> indexs in `CapacityScheduler#schedule` will always contains `start` index
> twice. This may not in line with the original intention and needs to be
> corrected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Szucs reassigned YARN-11685: -- Assignee: Peter Szucs > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839975#comment-17839975
]
ASF GitHub Bot commented on YARN-11684:
---
hadoop-yetus commented on PR #6753:
URL: https://github.com/apache/hadoop/pull/6753#issuecomment-2071566319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 01s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 86m 03s | | branch-3.4 passed |
| +1 :green_heart: | compile | 5m 45s | | branch-3.4 passed |
| +1 :green_heart: | checkstyle | 4m 34s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 5m 31s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 5m 13s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 140m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 06s | | the patch passed |
| +1 :green_heart: | javac | 3m 05s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 09s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 07s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 149m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 321m 10s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6753/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | asflicense | 5m 28s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6753/1/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 729m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore |
| | hadoop.yarn.server.resourcemanager.resource.TestResourceProfiles |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestFSSchedulerConfigurationStore
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCSAllocateCustomResource
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueConfigurationAutoRefreshPolicy
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigArgumentHandler
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigConverter
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigConverterMain
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.TestSignalContainer |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.volume.csi.TestVolumeProcessor |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppCustomResourceTypes
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsCustomResourceTypes
|
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesContainers |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839966#comment-17839966
]
ASF GitHub Bot commented on YARN-11191:
---
tomicooler commented on code in PR #6732:
URL: https://github.com/apache/hadoop/pull/6732#discussion_r1575731004
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##
@@ -1347,6 +1348,18 @@ public List getChildQueues() {
}
+ @Override
+ public List getChildQueuesByTryLock() {
+try {
+ while (!readLock.tryLock()){
+LockSupport.parkNanos(1);
Review Comment:
Ok, thanks. Fixed the checkstyle issues.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839932#comment-17839932
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2071402893
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 111m 14s | | trunk passed |
| +1 :green_heart: | compile | 6m 17s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 180m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 3m 06s | | the patch passed |
| +1 :green_heart: | javac | 3m 06s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 00s | | the patch passed |
| +1 :green_heart: | shadedclient | 191m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 7m 23s | | The patch does not
generate ASF License warnings. |
| | | 514m 05s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 686e2575880c 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / a2ee3f74b98a1dfada9090405ff3e14a922a166f |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6751/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6751/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update CpuResourceHandler implementation for cgroup v2 support
> --
>
> Key: YARN-11674
> URL: https://issues.apache.org/jira/browse/YARN-11674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about checking if
> CpuResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60]
> need any updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839923#comment-17839923
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2071323869
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 03s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 111m 35s | | trunk passed |
| +1 :green_heart: | compile | 6m 05s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 08s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 02s | | trunk passed |
| +1 :green_heart: | shadedclient | 173m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 01s | | the patch passed |
| +1 :green_heart: | javac | 3m 01s | | the patch passed |
| -1 :x: | blanks | 0m 01s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6760/1/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| +1 :green_heart: | checkstyle | 2m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 189m 08s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 18s | | The patch does not
generate ASF License warnings. |
| | | 503m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 7c8dbe3db292 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ecfd5504775d2173dfb2b894c73d361b5ddef831 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6760/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6760/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about checking if
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46]
> need any updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839899#comment-17839899
]
ASF GitHub Bot commented on YARN-11672:
---
slfan1989 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1575492286
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsHandler.java:
##
@@ -0,0 +1,579 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.SystemClock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractCGroupsHandler implements CGroupsHandler {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsHandler.class);
+ protected static final String MTAB_FILE = "/proc/mounts";
+
+ private final long deleteCGroupTimeout;
+ private final long deleteCGroupDelay;
+ private final Clock clock;
+
+ protected final String mtabFile;
+ protected final CGroupsMountConfig cGroupsMountConfig;
+ protected final ReadWriteLock rwLock;
+ protected Map controllerPaths;
+ protected Map> parsedMtab;
+ protected final PrivilegedOperationExecutor privilegedOperationExecutor;
+ protected final String cGroupPrefix;
+
+ /**
+ * Create cgroup handler object.
+ *
+ * @param confconfiguration
+ * @param privilegedOperationExecutor provides mechanisms to execute
+ *PrivilegedContainerOperations
+ * @param mtabmount file location
+ * @throws ResourceHandlerException if initialization failed
+ */
+ AbstractCGroupsHandler(Configuration conf, PrivilegedOperationExecutor
+ privilegedOperationExecutor, String mtab)
+ throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
+this.cGroupPrefix = conf.get(YarnConfiguration.
+NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
+.replaceAll("^/+", "").replaceAll("/+$", "");
+this.cGroupsMountConfig = new CGroupsMountConfig(conf);
+this.deleteCGroupTimeout = conf.getLong(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT) +
+conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
+YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) + 1000;
+this.deleteCGroupDelay =
+conf.getLong(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY);
+this.controllerPaths = new HashMap<>();
+this.parsedMtab = new HashMap<>();
+this.rwLock = new ReentrantReadWriteLock();
+this.privilegedOperationExecutor = privilegedOperationExecutor;
+
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839896#comment-17839896
]
ASF GitHub Bot commented on YARN-11684:
---
slfan1989 merged PR #6754:
URL: https://github.com/apache/hadoop/pull/6754
> PriorityQueueComparator violates general contract
> -
>
> Key: YARN-11684
> URL: https://issues.apache.org/jira/browse/YARN-11684
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.5.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1, 3.5.0
>
>
> YARN-10178 tried to fix the issue but there are still 2 property that might
> change during sorting which causes an exception.
> {code}
> 2024-04-10 12:36:56,420 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[Thread-28,5,main] threw an Exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeCollapse(TimSort.java:441)
> at java.util.TimSort.sort(TimSort.java:245)
> at java.util.Arrays.sort(Arrays.java:1512)
> at
> java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348)
> at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:260)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:1100)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:942)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1719)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1654)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1811)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1557)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:539)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:591)
> {code}
> The `queue.getAccessibleNodeLabels()` and `queue.getPriority()` could change
> in another thread while the `queues` are being sorted. Those should be saved
> when constructing the PriorityQueueResourcesForSorting helper object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11684) PriorityQueueComparator violates general contract
[
https://issues.apache.org/jira/browse/YARN-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839898#comment-17839898
]
ASF GitHub Bot commented on YARN-11684:
---
slfan1989 commented on PR #6754:
URL: https://github.com/apache/hadoop/pull/6754#issuecomment-2071158121
@tomicooler Thanks for the contribution!
> PriorityQueueComparator violates general contract
> -
>
> Key: YARN-11684
> URL: https://issues.apache.org/jira/browse/YARN-11684
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.5.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1, 3.5.0
>
>
> YARN-10178 tried to fix the issue but there are still 2 property that might
> change during sorting which causes an exception.
> {code}
> 2024-04-10 12:36:56,420 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[Thread-28,5,main] threw an Exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeCollapse(TimSort.java:441)
> at java.util.TimSort.sort(TimSort.java:245)
> at java.util.Arrays.sort(Arrays.java:1512)
> at
> java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:348)
> at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:260)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:1100)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:942)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1719)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1654)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1811)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1557)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:539)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:591)
> {code}
> The `queue.getAccessibleNodeLabels()` and `queue.getPriority()` could change
> in another thread while the `queues` are being sorted. Those should be saved
> when constructing the PriorityQueueResourcesForSorting helper object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839888#comment-17839888
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on code in PR #6732:
URL: https://github.com/apache/hadoop/pull/6732#discussion_r1575454919
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java:
##
@@ -1347,6 +1348,18 @@ public List getChildQueues() {
}
+ @Override
+ public List getChildQueuesByTryLock() {
+try {
+ while (!readLock.tryLock()){
+LockSupport.parkNanos(1);
Review Comment:
I have no other comments. we need to fix checkstyle, then we can merge this
pr.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839802#comment-17839802
]
ASF GitHub Bot commented on YARN-11674:
---
hadoop-yetus commented on PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2070385835
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 24 new + 10 unchanged - 2 fixed = 34 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 35s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 156m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6751 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 135ab82efafd 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a2ee3f74b98a1dfada9090405ff3e14a922a166f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6751/3/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839789#comment-17839789
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2070193014
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 33s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/1/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 72 new + 10 unchanged - 2 fixed = 82 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 36s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 111m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9edb0316bad3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ecfd5504775d2173dfb2b894c73d361b5ddef831 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839775#comment-17839775
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2070127772
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 57s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/16/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 16 new + 9 unchanged - 2 fixed = 25 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 43s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 173m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e5a3bc12acfb 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15636ffa943bcfde129862564bed322961f3be7b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/16/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839769#comment-17839769
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2070076468
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 27s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/15/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 16 new + 8 unchanged - 2 fixed = 24 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 52s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 174m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 84d5bf3ef590 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15636ffa943bcfde129862564bed322961f3be7b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/15/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839762#comment-17839762
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2070031217
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/13/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 9 unchanged - 2 fixed = 26 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 25s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 166m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b095f7d5ff76 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9106738c724c1e583bbdc2c65338b240b4a646d7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/13/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839764#comment-17839764
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2070035840
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 9 unchanged - 2 fixed = 26 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 37s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 166m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 05971d08205c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9106738c724c1e583bbdc2c65338b240b4a646d7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/14/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839726#comment-17839726
]
ASF GitHub Bot commented on YARN-11672:
---
brumi1024 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574902377
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsHandler.java:
##
@@ -0,0 +1,576 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.SystemClock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractCGroupsHandler implements CGroupsHandler {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsHandler.class);
+ protected static final String MTAB_FILE = "/proc/mounts";
+
+ private final long deleteCGroupTimeout;
+ private final long deleteCGroupDelay;
+ private final Clock clock;
+
+ protected final String mtabFile;
+ protected final CGroupsMountConfig cGroupsMountConfig;
+ protected final ReadWriteLock rwLock;
+ protected Map controllerPaths;
+ protected Map> parsedMtab;
+ protected final PrivilegedOperationExecutor privilegedOperationExecutor;
+ protected final String cGroupPrefix;
+
+ /**
+ * Create cgroup handler object.
+ *
+ * @param confconfiguration
+ * @param privilegedOperationExecutor provides mechanisms to execute
+ *PrivilegedContainerOperations
+ * @param mtabmount file location
+ * @throws ResourceHandlerException if initialization failed
+ */
+ AbstractCGroupsHandler(Configuration conf, PrivilegedOperationExecutor
+ privilegedOperationExecutor, String mtab)
+ throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
+this.cGroupPrefix = conf.get(YarnConfiguration.
+NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
+.replaceAll("^/+", "").replaceAll("/+$", "");
+this.cGroupsMountConfig = new CGroupsMountConfig(conf);
+this.deleteCGroupTimeout = conf.getLong(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT) +
+conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
+YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) + 1000;
+this.deleteCGroupDelay =
+conf.getLong(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY);
+this.controllerPaths = new HashMap<>();
+this.parsedMtab = new HashMap<>();
+this.rwLock = new ReentrantReadWriteLock();
+this.privilegedOperationExecutor = privilegedOperationExecutor;
+
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839725#comment-17839725 ] ASF GitHub Bot commented on YARN-11675: --- p-szucs opened a new pull request, #6760: URL: https://github.com/apache/hadoop/pull/6760 ### Description of PR Cgroup v2 has some changes in various controllers (some changed their functionality, some were removed). This task is about checking if MemoryResourceHandler's [implementation](https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46) need any updates. ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Update MemoryResourceHandler implementation for cgroup v2 support > - > > Key: YARN-11675 > URL: https://issues.apache.org/jira/browse/YARN-11675 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > MemoryResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11675: -- Labels: pull-request-available (was: ) > Update MemoryResourceHandler implementation for cgroup v2 support > - > > Key: YARN-11675 > URL: https://issues.apache.org/jira/browse/YARN-11675 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > MemoryResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839656#comment-17839656
]
ASF GitHub Bot commented on YARN-11672:
---
brumi1024 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574743317
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsV2HandlerImpl.java:
##
@@ -0,0 +1,270 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+/**
+ * Tests for the CGroups handler implementation.
+ */
+public class TestCGroupsV2HandlerImpl extends TestCGroupsHandlerBase {
+ // Create a controller file in the unified hierarchy of cgroup v2
+ @Override
+ protected String getControllerFilePath(String controllerName) {
+return new File(tmpPath, hierarchy).getAbsolutePath();
+ }
+
+ /*
+* Create a mock mtab file with the following content:
+* cgroup2 /path/to/parentDir cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
+*
+* Create the following cgroup v2 file hierarchy:
+* parentDir
+* ___
+* /\ \
+* cgroup.controllers cgroup.subtree_control
test-hadoop-yarn (hierarchyDir)
+*
_
+* /
\
+*
cgroup.controllers cgroup.subtree_control
+ */
+ public File createPremountedCgroups(File parentDir)
+ throws IOException {
+String baseCgroup2Line =
+"cgroup2 " + parentDir.getAbsolutePath()
++ " cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0\n";
+File mockMtab = createFileWithContent(parentDir,
UUID.randomUUID().toString(), baseCgroup2Line);
+
+String enabledControllers = "cpuset cpu io memory hugetlb pids rdma
misc\n";
+File controllersFile = createFileWithContent(parentDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE,
+enabledControllers);
+
+File subtreeControlFile = new File(parentDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE);
+Assert.assertTrue("empty subtree_control file should be created",
+subtreeControlFile.createNewFile());
+
+File hierarchyDir = new File(parentDir, hierarchy);
+if (!hierarchyDir.mkdirs()) {
+ String message = "Could not create directory " +
hierarchyDir.getAbsolutePath();
+ throw new IOException(message);
+}
+hierarchyDir.deleteOnExit();
+
+FileUtils.copyFile(controllersFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_CONTROLLERS_FILE));
+FileUtils.copyFile(subtreeControlFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE));
+
+return mockMtab;
+ }
+
+ @Test
+ public void testCGroupPaths() throws IOException, ResourceHandlerException {
+verifyZeroInteractions(privilegedOperationExecutorMock);
+File parentDir = new File(tmpPath);
+File mtab = createPremountedCgroups(parentDir);
+assertTrue("Sa
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839655#comment-17839655
]
ASF GitHub Bot commented on YARN-11672:
---
brumi1024 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574743251
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsV2HandlerImpl.java:
##
@@ -0,0 +1,270 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+/**
+ * Tests for the CGroups handler implementation.
+ */
+public class TestCGroupsV2HandlerImpl extends TestCGroupsHandlerBase {
+ // Create a controller file in the unified hierarchy of cgroup v2
+ @Override
+ protected String getControllerFilePath(String controllerName) {
+return new File(tmpPath, hierarchy).getAbsolutePath();
+ }
+
+ /*
+* Create a mock mtab file with the following content:
+* cgroup2 /path/to/parentDir cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
+*
+* Create the following cgroup v2 file hierarchy:
+* parentDir
+* ___
+* /\ \
+* cgroup.controllers cgroup.subtree_control
test-hadoop-yarn (hierarchyDir)
+*
_
+* /
\
+*
cgroup.controllers cgroup.subtree_control
+ */
+ public File createPremountedCgroups(File parentDir)
+ throws IOException {
+String baseCgroup2Line =
+"cgroup2 " + parentDir.getAbsolutePath()
++ " cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0\n";
+File mockMtab = createFileWithContent(parentDir,
UUID.randomUUID().toString(), baseCgroup2Line);
+
+String enabledControllers = "cpuset cpu io memory hugetlb pids rdma
misc\n";
+File controllersFile = createFileWithContent(parentDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE,
+enabledControllers);
+
+File subtreeControlFile = new File(parentDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE);
+Assert.assertTrue("empty subtree_control file should be created",
+subtreeControlFile.createNewFile());
+
+File hierarchyDir = new File(parentDir, hierarchy);
+if (!hierarchyDir.mkdirs()) {
+ String message = "Could not create directory " +
hierarchyDir.getAbsolutePath();
+ throw new IOException(message);
+}
+hierarchyDir.deleteOnExit();
+
+FileUtils.copyFile(controllersFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_CONTROLLERS_FILE));
+FileUtils.copyFile(subtreeControlFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE));
+
+return mockMtab;
+ }
+
+ @Test
+ public void testCGroupPaths() throws IOException, ResourceHandlerException {
+verifyZeroInteractions(privilegedOperationExecutorMock);
+File parentDir = new File(tmpPath);
+File mtab = createPremountedCgroups(parentDir);
+assertTrue("Sa
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839643#comment-17839643
]
ASF GitHub Bot commented on YARN-11672:
---
tomicooler commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574695343
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsV2HandlerImpl.java:
##
@@ -0,0 +1,270 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+/**
+ * Tests for the CGroups handler implementation.
+ */
+public class TestCGroupsV2HandlerImpl extends TestCGroupsHandlerBase {
+ // Create a controller file in the unified hierarchy of cgroup v2
+ @Override
+ protected String getControllerFilePath(String controllerName) {
+return new File(tmpPath, hierarchy).getAbsolutePath();
+ }
+
+ /*
+* Create a mock mtab file with the following content:
+* cgroup2 /path/to/parentDir cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
+*
+* Create the following cgroup v2 file hierarchy:
+* parentDir
+* ___
+* /\ \
+* cgroup.controllers cgroup.subtree_control
test-hadoop-yarn (hierarchyDir)
+*
_
+* /
\
+*
cgroup.controllers cgroup.subtree_control
+ */
+ public File createPremountedCgroups(File parentDir)
+ throws IOException {
+String baseCgroup2Line =
+"cgroup2 " + parentDir.getAbsolutePath()
++ " cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0\n";
+File mockMtab = createFileWithContent(parentDir,
UUID.randomUUID().toString(), baseCgroup2Line);
+
+String enabledControllers = "cpuset cpu io memory hugetlb pids rdma
misc\n";
+File controllersFile = createFileWithContent(parentDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE,
+enabledControllers);
+
+File subtreeControlFile = new File(parentDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE);
+Assert.assertTrue("empty subtree_control file should be created",
+subtreeControlFile.createNewFile());
+
+File hierarchyDir = new File(parentDir, hierarchy);
+if (!hierarchyDir.mkdirs()) {
+ String message = "Could not create directory " +
hierarchyDir.getAbsolutePath();
+ throw new IOException(message);
+}
+hierarchyDir.deleteOnExit();
+
+FileUtils.copyFile(controllersFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_CONTROLLERS_FILE));
+FileUtils.copyFile(subtreeControlFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE));
+
+return mockMtab;
+ }
+
+ @Test
+ public void testCGroupPaths() throws IOException, ResourceHandlerException {
+verifyZeroInteractions(privilegedOperationExecutorMock);
+File parentDir = new File(tmpPath);
+File mtab = createPremountedCgroups(parentDir);
+assertTrue("Sa
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839644#comment-17839644
]
ASF GitHub Bot commented on YARN-11672:
---
tomicooler commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574695897
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsV2HandlerImpl.java:
##
@@ -0,0 +1,270 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+/**
+ * Tests for the CGroups handler implementation.
+ */
+public class TestCGroupsV2HandlerImpl extends TestCGroupsHandlerBase {
+ // Create a controller file in the unified hierarchy of cgroup v2
+ @Override
+ protected String getControllerFilePath(String controllerName) {
+return new File(tmpPath, hierarchy).getAbsolutePath();
+ }
+
+ /*
+* Create a mock mtab file with the following content:
+* cgroup2 /path/to/parentDir cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
+*
+* Create the following cgroup v2 file hierarchy:
+* parentDir
+* ___
+* /\ \
+* cgroup.controllers cgroup.subtree_control
test-hadoop-yarn (hierarchyDir)
+*
_
+* /
\
+*
cgroup.controllers cgroup.subtree_control
+ */
+ public File createPremountedCgroups(File parentDir)
+ throws IOException {
+String baseCgroup2Line =
+"cgroup2 " + parentDir.getAbsolutePath()
++ " cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0\n";
+File mockMtab = createFileWithContent(parentDir,
UUID.randomUUID().toString(), baseCgroup2Line);
+
+String enabledControllers = "cpuset cpu io memory hugetlb pids rdma
misc\n";
+File controllersFile = createFileWithContent(parentDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE,
+enabledControllers);
+
+File subtreeControlFile = new File(parentDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE);
+Assert.assertTrue("empty subtree_control file should be created",
+subtreeControlFile.createNewFile());
+
+File hierarchyDir = new File(parentDir, hierarchy);
+if (!hierarchyDir.mkdirs()) {
+ String message = "Could not create directory " +
hierarchyDir.getAbsolutePath();
+ throw new IOException(message);
+}
+hierarchyDir.deleteOnExit();
+
+FileUtils.copyFile(controllersFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_CONTROLLERS_FILE));
+FileUtils.copyFile(subtreeControlFile, new File(hierarchyDir,
+CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE));
+
+return mockMtab;
+ }
+
+ @Test
+ public void testCGroupPaths() throws IOException, ResourceHandlerException {
+verifyZeroInteractions(privilegedOperationExecutorMock);
+File parentDir = new File(tmpPath);
+File mtab = createPremountedCgroups(parentDir);
+assertTrue("Sa
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839635#comment-17839635
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2069237515
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 26s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/12/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 9 unchanged - 2 fixed = 26 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 34s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 171m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1707299dcd62 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ec5647a20883679680d16af393c84e151128e30f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/12/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839634#comment-17839634
]
ASF GitHub Bot commented on YARN-11672:
---
hadoop-yetus commented on PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#issuecomment-2069237005
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/11/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 17 new + 9 unchanged - 2 fixed = 26 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 36s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 188m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9d3c1206de40 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ec5647a20883679680d16af393c84e151128e30f |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6734/11/testReport/ |
| Max. process+thread
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839617#comment-17839617
]
ASF GitHub Bot commented on YARN-11672:
---
tomicooler commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574578983
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsHandler.java:
##
@@ -0,0 +1,576 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.SystemClock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractCGroupsHandler implements CGroupsHandler {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsHandler.class);
+ protected static final String MTAB_FILE = "/proc/mounts";
+
+ private final long deleteCGroupTimeout;
+ private final long deleteCGroupDelay;
+ private final Clock clock;
+
+ protected final String mtabFile;
+ protected final CGroupsMountConfig cGroupsMountConfig;
+ protected final ReadWriteLock rwLock;
+ protected Map controllerPaths;
+ protected Map> parsedMtab;
+ protected final PrivilegedOperationExecutor privilegedOperationExecutor;
+ protected final String cGroupPrefix;
+
+ /**
+ * Create cgroup handler object.
+ *
+ * @param confconfiguration
+ * @param privilegedOperationExecutor provides mechanisms to execute
+ *PrivilegedContainerOperations
+ * @param mtabmount file location
+ * @throws ResourceHandlerException if initialization failed
+ */
+ AbstractCGroupsHandler(Configuration conf, PrivilegedOperationExecutor
+ privilegedOperationExecutor, String mtab)
+ throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
+this.cGroupPrefix = conf.get(YarnConfiguration.
+NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
+.replaceAll("^/+", "").replaceAll("/+$", "");
+this.cGroupsMountConfig = new CGroupsMountConfig(conf);
+this.deleteCGroupTimeout = conf.getLong(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT) +
+conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
+YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) + 1000;
+this.deleteCGroupDelay =
+conf.getLong(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY);
+this.controllerPaths = new HashMap<>();
+this.parsedMtab = new HashMap<>();
+this.rwLock = new ReentrantReadWriteLock();
+this.privilegedOperationExecutor = privilegedOperationExecutor;
+
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839590#comment-17839590
]
ASF GitHub Bot commented on YARN-11672:
---
tomicooler commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574502560
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsHandler.java:
##
@@ -0,0 +1,576 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.SystemClock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.PrintWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public abstract class AbstractCGroupsHandler implements CGroupsHandler {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsHandler.class);
+ protected static final String MTAB_FILE = "/proc/mounts";
+
+ private final long deleteCGroupTimeout;
+ private final long deleteCGroupDelay;
+ private final Clock clock;
+
+ protected final String mtabFile;
+ protected final CGroupsMountConfig cGroupsMountConfig;
+ protected final ReadWriteLock rwLock;
+ protected Map controllerPaths;
+ protected Map> parsedMtab;
+ protected final PrivilegedOperationExecutor privilegedOperationExecutor;
+ protected final String cGroupPrefix;
+
+ /**
+ * Create cgroup handler object.
+ *
+ * @param confconfiguration
+ * @param privilegedOperationExecutor provides mechanisms to execute
+ *PrivilegedContainerOperations
+ * @param mtabmount file location
+ * @throws ResourceHandlerException if initialization failed
+ */
+ AbstractCGroupsHandler(Configuration conf, PrivilegedOperationExecutor
+ privilegedOperationExecutor, String mtab)
+ throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
+this.cGroupPrefix = conf.get(YarnConfiguration.
+NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
+.replaceAll("^/+", "").replaceAll("/+$", "");
+this.cGroupsMountConfig = new CGroupsMountConfig(conf);
+this.deleteCGroupTimeout = conf.getLong(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_TIMEOUT) +
+conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
+YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) + 1000;
+this.deleteCGroupDelay =
+conf.getLong(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY,
+YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY);
+this.controllerPaths = new HashMap<>();
+this.parsedMtab = new HashMap<>();
+this.rwLock = new ReentrantReadWriteLock();
+this.privilegedOperationExecutor = privilegedOperationExecutor;
+
[jira] [Commented] (YARN-11686) Correct traversing indexs when scheduling asynchronously using Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839573#comment-17839573 ] ASF GitHub Bot commented on YARN-11686: --- onebox-li commented on PR #6750: URL: https://github.com/apache/hadoop/pull/6750#issuecomment-2068907134 Gentle ping @slfan1989 @zhuqi-lucas. PTAL if you have time, thanks. I think it's a minor list index change, there could be no need to introduce a new test. > Correct traversing indexs when scheduling asynchronously using Capacity > Scheduler > - > > Key: YARN-11686 > URL: https://issues.apache.org/jira/browse/YARN-11686 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Yihe Li >Priority: Minor > Labels: pull-request-available > > When scheduling asynchronously using Capacity Scheduler, the traversing > indexs in `CapacityScheduler#schedule` will always contains `start` index > twice. This may not in line with the original intention and needs to be > corrected. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11672) Create a CgroupHandler implementation for cgroup v2
[
https://issues.apache.org/jira/browse/YARN-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839564#comment-17839564
]
ASF GitHub Bot commented on YARN-11672:
---
brumi1024 commented on code in PR #6734:
URL: https://github.com/apache/hadoop/pull/6734#discussion_r1574392520
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsV2HandlerImpl.java:
##
@@ -0,0 +1,287 @@
+/*
+ * *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * /
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.verifyZeroInteractions;
+
+/**
+ * Tests for the CGroups handler implementation.
+ */
+public class TestCGroupsV2HandlerImpl extends TestCGroupsHandlerBase {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestCGroupsV2HandlerImpl.class);
+
+ // Create a controller file in the unified hierarchy of cgroup v2
+ @Override
+ protected String getControllerFilePath(String controllerName) {
+return new File(tmpPath, hierarchy).getAbsolutePath();
+ }
+
+ public File createPremountedCgroups(File parentDir)
+ throws IOException {
+// Format:
+// cgroup2 /sys/fs/cgroup cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
+String baseCgroup2Line =
+"cgroup2 " + parentDir.getAbsolutePath()
++ " cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0\n";
+
+File mockMtab = new File(parentDir, UUID.randomUUID().toString());
+if (!mockMtab.exists()) {
+ if (!mockMtab.createNewFile()) {
+String message = "Could not create file " + mockMtab.getAbsolutePath();
+throw new IOException(message);
+ }
+}
+FileWriter mtabWriter = new FileWriter(mockMtab.getAbsoluteFile());
+mtabWriter.write(baseCgroup2Line);
+mtabWriter.close();
+mockMtab.deleteOnExit();
+
+String enabledControllers = "cpuset cpu io memory hugetlb pids rdma
misc\n";
+
+File controllersFile = new File(parentDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE);
+FileWriter controllerWriter = new
FileWriter(controllersFile.getAbsoluteFile());
+controllerWriter.write(enabledControllers);
+controllerWriter.close();
+controllersFile.deleteOnExit();
+
+File subtreeControlFile = new File(parentDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE);
+Assert.assertTrue("empty subtree_control file should be created",
subtreeControlFile.createNewFile());
+
+File hierarchyDir = new File(parentDir, hierarchy);
+if (!hierarchyDir.mkdirs()) {
+ String message = "Could not create directory " +
hierarchyDir.getAbsolutePath();
+ throw new IOException(message);
+}
+hierarchyDir.deleteOnExit();
+FileUtils.copyFile(controllersFile, new File(hierarchyDir,
CGroupsHandler.CGROUP_CONTROLLERS_FILE));
+FileUtils.copyFile(subtreeControlFile, new File(hierarchyDir,
CGroupsHandler.CGROUP_SUBTREE_CONTROL_FILE));
+
+return mockMtab;
+ }
+
+ @Test
+ public void testCGroupPaths() throws IOException {
+verifyZeroInteractions(privilegedOperationExecutorMock);
+CGroupsHandler cGroupsHandler = null;
+File parentDir = new File(tmpPath);
+File mtab = createPremountedCgroups(parentDir);
+assertTru
