[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793117#comment-17793117
]
wangzhihui commented on YARN-11622:
---
[~hexiaoqiao]
[~elgoiri] [~slfan1989] Thank you all, I will start the relevant repairs soon.
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh the Scheduler
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793106#comment-17793106
]
Xiaoqiao He commented on YARN-11622:
Great, thanks [~slfan1989] and [~elgoiri].
[~hiwangzhihui] would you mind to try submit PR via GitHub, we will follow and
move this bugfix forwards.
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the
[jira] [Commented] (YARN-11498) Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml
[
https://issues.apache.org/jira/browse/YARN-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793091#comment-17793091
]
ASF GitHub Bot commented on YARN-11498:
---
hadoop-yetus commented on PR #6063:
URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1839920406
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 14m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 42m 1s | | branch-3.3 passed |
| +1 :green_heart: | compile | 21m 9s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 9m 20s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 6m 38s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 135m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 26m 18s | | the patch passed |
| +1 :green_heart: | compile | 20m 20s | | the patch passed |
| +1 :green_heart: | javac | 20m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 8m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 7m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 60m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 19m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 5m 12s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 23m 51s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 4m 44s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 99m 48s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | |
hadoop-yarn-applications-catalog-webapp in the patch passed. |
| +1 :green_heart: | unit | 0m 58s | | hadoop-resourceestimator in the
patch passed. |
| +1 :green_heart: | unit | 0m 40s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 42s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 407m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6063 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux dbed5e4c8923 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 66cede4ab9caa940ecf06850563ad5fb10c30b5f |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/10/testReport/ |
| Max. process+thread count | 1271 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793035#comment-17793035
]
Íñigo Goiri commented on YARN-11622:
Not having a single place to track the locks is obviously an issue.
Adding this entity tracking all the access makes sense to me.
The onyl concern for me would be performance, let's add some evaluation for
that once we have the implementation.
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred
[jira] [Commented] (YARN-11613) [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues.
[
https://issues.apache.org/jira/browse/YARN-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793012#comment-17793012
]
ASF GitHub Bot commented on YARN-11613:
---
hadoop-yetus commented on PR #6295:
URL: https://github.com/apache/hadoop/pull/6295#issuecomment-1839323160
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 41s | | trunk passed |
| +1 :green_heart: | compile | 7m 10s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 6m 34s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 8s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 49s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 23s | | the patch passed |
| +1 :green_heart: | compile | 6m 41s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 6m 41s | | the patch passed |
| +1 :green_heart: | javac | 6m 41s | | the patch passed |
| +1 :green_heart: | compile | 6m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | cc | 6m 34s | | the patch passed |
| +1 :green_heart: | javac | 6m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 56s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 33s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 10m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 48s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 1s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 101m 8s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 27s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 42s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 344m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6295/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6295 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux f0ef532d2868
[jira] [Updated] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-11622:
---
Summary: ResourceManager asynchronous switch from Standy to Active
exception (was: ResourceManager asynchronous switch to Standy、Active exception)
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh the
[jira] [Commented] (YARN-11619) [Federation] Router CLI Supports List SubClusters.
[ https://issues.apache.org/jira/browse/YARN-11619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792854#comment-17792854 ] ASF GitHub Bot commented on YARN-11619: --- slfan1989 commented on PR #6304: URL: https://github.com/apache/hadoop/pull/6304#issuecomment-1838663667 @goiri Can you help reiew this pr? Thank you very much! > [Federation] Router CLI Supports List SubClusters. > -- > > Key: YARN-11619 > URL: https://issues.apache.org/jira/browse/YARN-11619 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > We need to support list subcluster on the command line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792850#comment-17792850
]
Shilun Fan commented on YARN-11622:
---
[~hiwangzhihui] Thank you for reporting this issue, I will continue to follow
up on this issue. cc: [~hexiaoqiao]
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh the Scheduler
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792839#comment-17792839
]
Xiaoqiao He commented on YARN-11622:
Great catch here! After the logic review, I think IT IS one HA issue. cc
[~slfan1989], [~elgoiri] Would you mind to have another check?
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792813#comment-17792813
]
wangzhihui commented on YARN-11622:
---
The root cause of this issue can be traced back to the asynchronous processing
logic introduced in the PATCH of the 3.0.0-alpha4 branch.
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to
[jira] [Updated] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangzhihui updated YARN-11622:
--
Affects Version/s: 3.3.0
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh the Scheduler configuration. At this time, the
> csConfProvider property of the CapacityScheduler is not initialized and its
> value is null.
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792812#comment-17792812
]
wangzhihui commented on YARN-11622:
---
hi ,[~hexiaoqiao] I have checked the Active Branch 3.4, 3.3, 3.2, and the
latest 3.3.6 versions, and they all have the same issue.
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh
[jira] [Updated] (YARN-11622) ResourceManager asynchronous switch to Standy、Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangzhihui updated YARN-11622:
--
Affects Version/s: 3.1.1
3.0.0-alpha4
(was: 3.0.0)
(was: 3.1.3)
> ResourceManager asynchronous switch to Standy、Active exception
> --
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1
>Reporter: wangzhihui
>Priority: Major
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore triggered toActive event and
> toStandby event at 06:17:35, Two asynchronous events are respectively
> referred to as Thread_ 1、Thread_ 2.
> * During the execution of Thread_ 1 the CapacityScheduler.reinitialize is
> called to refresh the Scheduler configuration. At
[jira] [Commented] (YARN-11613) [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues.
[
https://issues.apache.org/jira/browse/YARN-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792790#comment-17792790
]
ASF GitHub Bot commented on YARN-11613:
---
hadoop-yetus commented on PR #6295:
URL: https://github.com/apache/hadoop/pull/6295#issuecomment-1838451734
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 25s | | trunk passed |
| +1 :green_heart: | compile | 7m 53s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 7s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 50s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 21s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 10s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 7m 10s | | the patch passed |
| +1 :green_heart: | javac | 7m 10s | | the patch passed |
| +1 :green_heart: | compile | 6m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | cc | 7m 0s | | the patch passed |
| +1 :green_heart: | javac | 6m 59s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6295/15/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 53s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6295/15/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 67 unchanged -
0 fixed = 68 total (was 67) |
| +1 :green_heart: | mvnsite | 4m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 39s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 20s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 10m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 12s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 34s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 54s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 104m 49s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 19s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 42s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 366m 52s | | |
[jira] [Updated] (YARN-11624) CapacityScheduler: Add configuration to disable AM preemption
[ https://issues.apache.org/jira/browse/YARN-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanbin.zhang updated YARN-11624: Description: Disable AM-preemption for CapacityScheduler, like FairScheduler: -YARN-9537- (was: Disable AM-preemption for CapacityScheduler like fair-scheduler: -YARN-9537-) > CapacityScheduler: Add configuration to disable AM preemption > - > > Key: YARN-11624 > URL: https://issues.apache.org/jira/browse/YARN-11624 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: yanbin.zhang >Priority: Major > > Disable AM-preemption for CapacityScheduler, like FairScheduler: -YARN-9537- -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11624) CapacityScheduler: Add configuration to disable AM preemption
[ https://issues.apache.org/jira/browse/YARN-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanbin.zhang updated YARN-11624: Description: Like FairScheduler feature: YARN-9537, for CapacityScheduler to disable AM-preemption. (was: Like FairScheduler feature: YARN-9537, add global flag for CapacityScheduler to disable AM-preemption.) > CapacityScheduler: Add configuration to disable AM preemption > - > > Key: YARN-11624 > URL: https://issues.apache.org/jira/browse/YARN-11624 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: yanbin.zhang >Priority: Major > > Like FairScheduler feature: YARN-9537, for CapacityScheduler to disable > AM-preemption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11624) CapacityScheduler: Add configuration to disable AM preemption
[ https://issues.apache.org/jira/browse/YARN-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanbin.zhang updated YARN-11624: Description: Like FairScheduler feature: YARN-9537, add global flag for CapacityScheduler to disable AM-preemption. (was: Like FairScheduler feature: YARN-10625, add global flag for CapacityScheduler to disable AM-preemption.) > CapacityScheduler: Add configuration to disable AM preemption > - > > Key: YARN-11624 > URL: https://issues.apache.org/jira/browse/YARN-11624 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: yanbin.zhang >Priority: Major > > Like FairScheduler feature: YARN-9537, add global flag for CapacityScheduler > to disable AM-preemption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11624) CapacityScheduler: Add configuration to disable AM preemption
[ https://issues.apache.org/jira/browse/YARN-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanbin.zhang updated YARN-11624: Summary: CapacityScheduler: Add configuration to disable AM preemption (was: CapacityScheduler: add global flag to disable AM-preemption) > CapacityScheduler: Add configuration to disable AM preemption > - > > Key: YARN-11624 > URL: https://issues.apache.org/jira/browse/YARN-11624 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: yanbin.zhang >Priority: Major > > Like FairScheduler feature: YARN-10625, add global flag for CapacityScheduler > to disable AM-preemption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11624) CapacityScheduler: add global flag to disable AM-preemption
yanbin.zhang created YARN-11624: --- Summary: CapacityScheduler: add global flag to disable AM-preemption Key: YARN-11624 URL: https://issues.apache.org/jira/browse/YARN-11624 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: yanbin.zhang Like FairScheduler feature: YARN-10625, add global flag for CapacityScheduler to disable AM-preemption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11115) Add configuration to disable AM preemption for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792755#comment-17792755 ] yanbin.zhang commented on YARN-5: - Take it up. > Add configuration to disable AM preemption for capacity scheduler > - > > Key: YARN-5 > URL: https://issues.apache.org/jira/browse/YARN-5 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Yuan Luo >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > I think it's necessary to add configuration to disable AM preemption for > capacity-scheduler, like fair-scheduler feature: YARN-9537. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11623) FairScheduler: Document AM preemption related changes (YARN-9537 and YARN-10625)
[
https://issues.apache.org/jira/browse/YARN-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792732#comment-17792732
]
ASF GitHub Bot commented on YARN-11623:
---
hadoop-yetus commented on PR #6320:
URL: https://github.com/apache/hadoop/pull/6320#issuecomment-1838099855
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 80m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6320/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6320 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint |
| uname | Linux 0e8eb0ac68fa 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 47f8cc8b66373bc6e4caecf04e317c72749bf19f |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6320/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> FairScheduler: Document AM preemption related changes (YARN-9537 and
> YARN-10625)
>
>
> Key: YARN-11623
> URL: https://issues.apache.org/jira/browse/YARN-11623
> Project: Hadoop YARN
> Issue Type: Task
> Components: fairscheduler
>Reporter: yanbin.zhang
>Priority: Major
> Labels: pull-request-available
>
> Extend the documentation with these enhancements about YARN-9537 and
> YARN-10625.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
