[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Description: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. For example: When adding a dynamic queue, the other children's max app of parent queue are not updated correctly: !image-2021-02-20-15-53-51-099.png|width=639,height=509! The new added queue's max app will updated correctly: !image-2021-02-20-15-55-44-780.png|width=542,height=426! was: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. For example: When adding a dynamic queue, the other children's max app of parent queue are not updated correctly: !image-2021-02-20-15-53-51-099.png|width=639,height=509! > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png, > image-2021-02-20-15-53-51-099.png, image-2021-02-20-15-55-44-780.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. > For example: > When adding a dynamic queue, the other children's max app of parent queue are > not updated correctly: > !image-2021-02-20-15-53-51-099.png|width=639,height=509! > The new added queue's max app will updated correctly: > !image-2021-02-20-15-55-44-780.png|width=542,height=426! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Attachment: image-2021-02-20-15-55-44-780.png > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png, > image-2021-02-20-15-53-51-099.png, image-2021-02-20-15-55-44-780.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. > For example: > When adding a dynamic queue, the other children's max app of parent queue are > not updated correctly: > !image-2021-02-20-15-53-51-099.png|width=639,height=509! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Description: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. For example: When adding a dynamic queue, the other children's max app of parent queue are not updated correctly: !image-2021-02-20-15-53-51-099.png|width=639,height=509! was: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. When add dynamic queue: The other children's max app of parent queue are not updated correctly: !image-2021-02-20-15-49-58-677.png|width=729,height=478! > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png, > image-2021-02-20-15-53-51-099.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. > For example: > When adding a dynamic queue, the other children's max app of parent queue are > not updated correctly: > !image-2021-02-20-15-53-51-099.png|width=639,height=509! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Description: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. When add dynamic queue: The other children's max app of parent queue are not updated correctly: !image-2021-02-20-15-49-58-677.png|width=729,height=478! was: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png, > image-2021-02-20-15-53-51-099.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. > When add dynamic queue: > The other children's max app of parent queue are not updated correctly: > !image-2021-02-20-15-49-58-677.png|width=729,height=478! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Attachment: image-2021-02-20-15-53-51-099.png > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png, > image-2021-02-20-15-53-51-099.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Attachment: image-2021-02-20-15-49-58-677.png > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-15-49-58-677.png > > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10641) Fix maxApllications update error when add new queues.
[ https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10641: -- Description: When refactor the update logic in YARN-10504 . The update max applications based abs/cap is wrong, this should be fixed, because the max applications is key part to limit applications in CS. > Fix maxApllications update error when add new queues. > - > > Key: YARN-10641 > URL: https://issues.apache.org/jira/browse/YARN-10641 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > When refactor the update logic in YARN-10504 . > The update max applications based abs/cap is wrong, this should be fixed, > because the max applications is key part to limit applications in CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10641) Fix maxApllications update error when add new queues.
Qi Zhu created YARN-10641: - Summary: Fix maxApllications update error when add new queues. Key: YARN-10641 URL: https://issues.apache.org/jira/browse/YARN-10641 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287559#comment-17287559 ] Qi Zhu commented on YARN-10640: --- cc [~snemeth] [~gandras] [~bteke] I have updated a patch, and confirmed in my local cluster, could you help review it? Thanks. !image-2021-02-20-14-18-56-261.png! !image-2021-02-20-14-19-30-767.png|width=489,height=342! > Ajust the queue Configured capacity to Configured weight number for weight > mode in UI. > --- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, image-2021-02-20-11-21-50-306.png, > image-2021-02-20-14-18-56-261.png, image-2021-02-20-14-19-30-767.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Attachment: image-2021-02-20-14-19-30-767.png > Ajust the queue Configured capacity to Configured weight number for weight > mode in UI. > --- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, image-2021-02-20-11-21-50-306.png, > image-2021-02-20-14-18-56-261.png, image-2021-02-20-14-19-30-767.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Attachment: image-2021-02-20-14-18-56-261.png > Ajust the queue Configured capacity to Configured weight number for weight > mode in UI. > --- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, image-2021-02-20-11-21-50-306.png, > image-2021-02-20-14-18-56-261.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Summary: Ajust the queue Configured capacity to Configured weight number for weight mode in UI. (was: Ajust the queue Configured capacity to weight number for weight mode in UI.) > Ajust the queue Configured capacity to Configured weight number for weight > mode in UI. > --- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue Configured capacity to weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Summary: Ajust the queue Configured capacity to weight number for weight mode in UI. (was: Ajust the queue capacity to weight number for weight mode in UI.) > Ajust the queue Configured capacity to weight number for weight mode in UI. > > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Description: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0. I think this should change to Configured Weight if we use weight mode, this will be helpful. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! was: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0. I think this should change to Configured Weight if we use weight mode. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! > Ajust the queue capacity to weight number for weight mode in UI. > - > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Description: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0. I think this should change to Configured Weight if we use weight mode. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! was: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0, it will make confused, because the absolute will all show the right Configured Capacity, we should fix it. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! > Ajust the queue capacity to weight number for weight mode in UI. > - > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Summary: Ajust the queue capacity to weight number for weight mode in UI. (was: Ajust the queue capacity to weight mode in UI.) > Ajust the queue capacity to weight number for weight mode in UI. > - > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0, it will make confused, because the absolute will > all show the right Configured Capacity, we should fix it. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Description: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0, it will make confused, because the absolute will all show the right Configured Capacity, we should fix it. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! was: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0, it will make confused, because the absolute/cap will all show the right Configured Capacity, we should fix it. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! > Ajust the queue capacity to weight mode in UI. > -- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0, it will make confused, because the absolute will > all show the right Configured Capacity, we should fix it. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287537#comment-17287537
]
Qi Zhu commented on YARN-10639:
---
Fixed test and checkstyle.
> Queueinfo related capacity, should ajusted to weight mode.
> --
>
> Key: YARN-10639
> URL: https://issues.apache.org/jira/browse/YARN-10639
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10639.001.patch, YARN-10639.002.patch
>
>
> {color:#172b4d}The class QueueInfo capacity field should consider the weight
> mode.{color}
> {color:#172b4d}Now when client use getQueueInfo to get queue capacity in
> weight mode, i always return 0, it is wrong.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qi Zhu updated YARN-10639:
--
Attachment: YARN-10639.002.patch
> Queueinfo related capacity, should ajusted to weight mode.
> --
>
> Key: YARN-10639
> URL: https://issues.apache.org/jira/browse/YARN-10639
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10639.001.patch, YARN-10639.002.patch
>
>
> {color:#172b4d}The class QueueInfo capacity field should consider the weight
> mode.{color}
> {color:#172b4d}Now when client use getQueueInfo to get queue capacity in
> weight mode, i always return 0, it is wrong.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287530#comment-17287530
]
Hadoop QA commented on YARN-10639:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 46s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/646/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 288 unchanged - 0 fixed = 289 total (was 288)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 38s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Description: In weight mode: Both the weight mode static queue and the dynamic queue will show the Configured Capacity to 0, it will make confused, because the absolute/cap will all show the right Configured Capacity, we should fix it. Such as in dynamic weight mode queue: !image-2021-02-20-11-21-50-306.png|width=528,height=374! > Ajust the queue capacity to weight mode in UI. > -- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0, it will make confused, because the absolute/cap > will all show the right Configured Capacity, we should fix it. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10640) Ajust the queue capacity to weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10640: -- Attachment: image-2021-02-20-11-21-50-306.png > Ajust the queue capacity to weight mode in UI. > -- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: image-2021-02-20-11-21-50-306.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10640) Ajust the queue capacity to weight mode in UI.
Qi Zhu created YARN-10640: - Summary: Ajust the queue capacity to weight mode in UI. Key: YARN-10640 URL: https://issues.apache.org/jira/browse/YARN-10640 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287461#comment-17287461
]
caozhiqiang edited comment on YARN-10501 at 2/20/21, 2:52 AM:
--
Thank you. OK，I will put up a 2.10 patch later. If it need to create a new
patch for 2.10 patch? I don't know this process.
was (Author: caozhiqiang):
Thank you. OK，I will put up a 2.10 patch later. If it need to create a new
patch for 2.10 patch? I don't no
> Can't remove all node labels after add node label without nodemanager port
> --
>
> Key: YARN-10501
> URL: https://issues.apache.org/jira/browse/YARN-10501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Critical
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10501.002.patch, YARN-10501.003.patch,
> YARN-10501.004.patch
>
>
> When add a label to nodes without nodemanager port or use WILDCARD_PORT (0)
> port, it can't remove all label info in these nodes
> Reproduce process:
> {code:java}
> 1.yarn rmadmin -addToClusterNodeLabels "cpunode(exclusive=true)"
> 2.yarn rmadmin -replaceLabelsOnNode "server001=cpunode"
> 3.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":["server001:0","server001:45454"],"partitionInfo":{"resourceAvailable":{"memory":"510","vCores":"1","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"510"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"1"}]}}}
> 4.yarn rmadmin -replaceLabelsOnNode "server001"
> 5.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":"server001:45454","partitionInfo":{"resourceAvailable":{"memory":"0","vCores":"0","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"0"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"0"}]}}}
> {code}
> You can see after the 4 process to remove nodemanager labels, the label info
> is still in the node info.
> {code:java}
> 641 case REPLACE:
> 642 replaceNodeForLabels(nodeId, host.labels, labels);
> 643 replaceLabelsForNode(nodeId, host.labels, labels);
> 644 host.labels.clear();
> 645 host.labels.addAll(labels);
> 646 for (Node node : host.nms.values()) {
> 647 replaceNodeForLabels(node.nodeId, node.labels, labels);
> 649 node.labels = null;
> 650 }
> 651 break;{code}
> The cause is in 647 line, when add labels to node without port, the 0 port
> and the real nm port with be both add to node info, and when remove labels,
> the parameter node.labels in 647 line is null, so it will not remove the old
> label.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287461#comment-17287461
]
caozhiqiang commented on YARN-10501:
Thank you. OK，I will put up a 2.10 patch later. If it need to create a new
patch for 2.10 patch? I don't no
> Can't remove all node labels after add node label without nodemanager port
> --
>
> Key: YARN-10501
> URL: https://issues.apache.org/jira/browse/YARN-10501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Critical
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10501.002.patch, YARN-10501.003.patch,
> YARN-10501.004.patch
>
>
> When add a label to nodes without nodemanager port or use WILDCARD_PORT (0)
> port, it can't remove all label info in these nodes
> Reproduce process:
> {code:java}
> 1.yarn rmadmin -addToClusterNodeLabels "cpunode(exclusive=true)"
> 2.yarn rmadmin -replaceLabelsOnNode "server001=cpunode"
> 3.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":["server001:0","server001:45454"],"partitionInfo":{"resourceAvailable":{"memory":"510","vCores":"1","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"510"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"1"}]}}}
> 4.yarn rmadmin -replaceLabelsOnNode "server001"
> 5.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":"server001:45454","partitionInfo":{"resourceAvailable":{"memory":"0","vCores":"0","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"0"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"0"}]}}}
> {code}
> You can see after the 4 process to remove nodemanager labels, the label info
> is still in the node info.
> {code:java}
> 641 case REPLACE:
> 642 replaceNodeForLabels(nodeId, host.labels, labels);
> 643 replaceLabelsForNode(nodeId, host.labels, labels);
> 644 host.labels.clear();
> 645 host.labels.addAll(labels);
> 646 for (Node node : host.nms.values()) {
> 647 replaceNodeForLabels(node.nodeId, node.labels, labels);
> 649 node.labels = null;
> 650 }
> 651 break;{code}
> The cause is in 647 line, when add labels to node without port, the 0 port
> and the real nm port with be both add to node info, and when remove labels,
> the parameter node.labels in 647 line is null, so it will not remove the old
> label.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287460#comment-17287460
]
Qi Zhu commented on YARN-10639:
---
cc [~snemeth] [~gandras] [~bteke]
Could you help review it, now in weight mode, the user can't use getQueueInfo
to get capacity correctly, i have fixed it in this issue.
Thanks.
> Queueinfo related capacity, should ajusted to weight mode.
> --
>
> Key: YARN-10639
> URL: https://issues.apache.org/jira/browse/YARN-10639
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10639.001.patch
>
>
> {color:#172b4d}The class QueueInfo capacity field should consider the weight
> mode.{color}
> {color:#172b4d}Now when client use getQueueInfo to get queue capacity in
> weight mode, i always return 0, it is wrong.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qi Zhu updated YARN-10639:
--
Description:
{color:#172b4d}The class QueueInfo capacity field should consider the weight
mode.{color}
{color:#172b4d}Now when client use getQueueInfo to get queue capacity in weight
mode, i always return 0, it is wrong.{color}
was:{color:#172b4d}The class QueueInfo capacity field should consider the
weight mode.{color}
> Queueinfo related capacity, should ajusted to weight mode.
> --
>
> Key: YARN-10639
> URL: https://issues.apache.org/jira/browse/YARN-10639
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10639.001.patch
>
>
> {color:#172b4d}The class QueueInfo capacity field should consider the weight
> mode.{color}
> {color:#172b4d}Now when client use getQueueInfo to get queue capacity in
> weight mode, i always return 0, it is wrong.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10632) Make maximum depth allowed configurable.
[ https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287455#comment-17287455 ] Qi Zhu commented on YARN-10632: --- Thanks [~shuzirra] for reminder, i will take it into consideration when we revisit this. > Make maximum depth allowed configurable. > > > Key: YARN-10632 > URL: https://issues.apache.org/jira/browse/YARN-10632 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10632.001.patch > > > Now the max depth allowed are fixed to 2. But i think this should be > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[ https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287453#comment-17287453 ] Qi Zhu commented on YARN-10623: --- cc [~bteke] [~ztang] [~shuzirra] [~pbacsko] [~snemeth] Could you help review it, when you are free? Thanks. > Capacity scheduler should support refresh queue automatically by a thread > policy. > - > > Key: YARN-10623 > URL: https://issues.apache.org/jira/browse/YARN-10623 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10623.001.patch, YARN-10623.002.patch, > YARN-10623.003.patch > > > In fair scheduler, it is supported that refresh queue related conf > automatically by a thread to reload, but in capacity scheduler we only > support to refresh queue related changes by refreshQueues, it is needed for > our cluster to realize queue manage. > cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[ https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287453#comment-17287453 ] Qi Zhu edited comment on YARN-10623 at 2/20/21, 2:14 AM: - cc [~bteke] [~gandras] [~ztang] [~shuzirra] [~pbacsko] [~snemeth] Could you help review it, when you are free? Thanks. was (Author: zhuqi): cc [~bteke] [~ztang] [~shuzirra] [~pbacsko] [~snemeth] Could you help review it, when you are free? Thanks. > Capacity scheduler should support refresh queue automatically by a thread > policy. > - > > Key: YARN-10623 > URL: https://issues.apache.org/jira/browse/YARN-10623 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10623.001.patch, YARN-10623.002.patch, > YARN-10623.003.patch > > > In fair scheduler, it is supported that refresh queue related conf > automatically by a thread to reload, but in capacity scheduler we only > support to refresh queue related changes by refreshQueues, it is needed for > our cluster to realize queue manage. > cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-10501:
---
Fix Version/s: 3.2.3
3.1.5
3.3.1
3.4.0
Thanks for the updated patch, [~caozhiqiang] I fixed up the small checkstyle
warning and committed this patch to trunk (3.4), branch-3.3, branch-3.2, and
branch-3.1. There is a conflict in branch-2.10. If you'd like it to be pulled
back there as well, please put up a 2.10 patch.
> Can't remove all node labels after add node label without nodemanager port
> --
>
> Key: YARN-10501
> URL: https://issues.apache.org/jira/browse/YARN-10501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Critical
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10501.002.patch, YARN-10501.003.patch,
> YARN-10501.004.patch
>
>
> When add a label to nodes without nodemanager port or use WILDCARD_PORT (0)
> port, it can't remove all label info in these nodes
> Reproduce process:
> {code:java}
> 1.yarn rmadmin -addToClusterNodeLabels "cpunode(exclusive=true)"
> 2.yarn rmadmin -replaceLabelsOnNode "server001=cpunode"
> 3.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":["server001:0","server001:45454"],"partitionInfo":{"resourceAvailable":{"memory":"510","vCores":"1","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"510"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"1"}]}}}
> 4.yarn rmadmin -replaceLabelsOnNode "server001"
> 5.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":"server001:45454","partitionInfo":{"resourceAvailable":{"memory":"0","vCores":"0","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"0"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"0"}]}}}
> {code}
> You can see after the 4 process to remove nodemanager labels, the label info
> is still in the node info.
> {code:java}
> 641 case REPLACE:
> 642 replaceNodeForLabels(nodeId, host.labels, labels);
> 643 replaceLabelsForNode(nodeId, host.labels, labels);
> 644 host.labels.clear();
> 645 host.labels.addAll(labels);
> 646 for (Node node : host.nms.values()) {
> 647 replaceNodeForLabels(node.nodeId, node.labels, labels);
> 649 node.labels = null;
> 650 }
> 651 break;{code}
> The cause is in 647 line, when add labels to node without port, the 0 port
> and the real nm port with be both add to node info, and when remove labels,
> the parameter node.labels in 647 line is null, so it will not remove the old
> label.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287339#comment-17287339
]
Hadoop QA commented on YARN-10627:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 41s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
|
[jira] [Commented] (YARN-10632) Make maximum depth allowed configurable.
[ https://issues.apache.org/jira/browse/YARN-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287256#comment-17287256 ] Gergely Pollak commented on YARN-10632: --- Also please note that this has an effect on the placement engine as well, and currently the path validation methods are separated, so even if CS accepts deeper queue creations, placement rules wouldn't. First we need to centralize the validation for the queue paths to be created (YARN-10584), and also we need to make a few adjustments in the placement engine in order to make this work. > Make maximum depth allowed configurable. > > > Key: YARN-10632 > URL: https://issues.apache.org/jira/browse/YARN-10632 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10632.001.patch > > > Now the max depth allowed are fixed to 2. But i think this should be > configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287213#comment-17287213
]
Benjamin Teke commented on YARN-10627:
--
[~zhuqi], indeed, thanks for finding it. Fixed it with the second patch, will
introduce some tests for it in patch 003.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [=
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.002.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " + aclsString
>
[jira] [Commented] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287169#comment-17287169
]
Hadoop QA commented on YARN-10501:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 20s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
53s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/643/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch
generated 1 new + 52 unchanged - 0 fixed = 53 total (was 52) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 48s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
[jira] [Updated] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
[
https://issues.apache.org/jira/browse/YARN-10639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qi Zhu updated YARN-10639:
--
Description: {color:#172b4d}The class QueueInfo capacity field should
consider the weight mode.{color}
> Queueinfo related capacity, should ajusted to weight mode.
> --
>
> Key: YARN-10639
> URL: https://issues.apache.org/jira/browse/YARN-10639
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
>
> {color:#172b4d}The class QueueInfo capacity field should consider the weight
> mode.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287152#comment-17287152
]
Qi Zhu edited comment on YARN-10627 at 2/19/21, 4:18 PM:
-
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
Only the weight is -1, we can make sure it is not weight mode.
So the code may change to, in leafqueue:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
In abstractCSQueue:
{code:java}
protected String getCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight=" + queueCapacities.getWeight() + ", " +
"normalizedWeight=" + queueCapacities.getNormalizedWeight();
} else {
return "capacity=" + queueCapacities.getCapacity();
}
}
{code}
And we can add a test to confirm it.
Thanks.
was (Author: zhuqi):
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
Only the weight is -1, we can make sure it is not weight mode.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
Thanks.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
>
[jira] [Created] (YARN-10639) Queueinfo related capacity, should ajusted to weight mode.
Qi Zhu created YARN-10639: - Summary: Queueinfo related capacity, should ajusted to weight mode. Key: YARN-10639 URL: https://issues.apache.org/jira/browse/YARN-10639 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287152#comment-17287152
]
Qi Zhu edited comment on YARN-10627 at 2/19/21, 4:12 PM:
-
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
Only the weight is -1, we can make sure it is not weight mode.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
Thanks.
was (Author: zhuqi):
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
Thanks.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" +
[jira] [Comment Edited] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287152#comment-17287152
]
Qi Zhu edited comment on YARN-10627 at 2/19/21, 4:11 PM:
-
[~bteke] [~gandras] [~snemeth]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
Thanks.
was (Author: zhuqi):
[~bteke] [~gandras]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [=
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287152#comment-17287152
]
Qi Zhu commented on YARN-10627:
---
[~bteke] [~gandras]
!image-2021-02-20-00-07-09-875.png|width=420,height=156!
When i deep into the code, and look the ui, when weight setted, the capacity
will be 0.
So the code may change to, such as:
{code:java}
private String getExtendedCapacityOrWeightString() {
if (queueCapacities.getWeight() != -1) {
return "weight = " + queueCapacities.getWeight()
+ " [= (float) configuredCapacity (with w suffix)] " + "\n"
+ "normalizedWeight = " + queueCapacities.getNormalizedWeight()
+ " [= (float) configuredCapacity / sum(configuredCapacity of " +
"all queues under the parent)]";
} else {
return "capacity = " + queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]";
}
}
{code}
And we can add a test to confirm it.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
>
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qi Zhu updated YARN-10627:
--
Attachment: image-2021-02-20-00-07-09-875.png
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " + aclsString
> + " [= configuredAcls
[jira] [Commented] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287119#comment-17287119
]
caozhiqiang commented on YARN-10501:
Thank you for your comments. Yes, I make a mistake in unit test that it does
not test the functionality. I have rewrite to test this case. And I also modify
the exception info.
> Can't remove all node labels after add node label without nodemanager port
> --
>
> Key: YARN-10501
> URL: https://issues.apache.org/jira/browse/YARN-10501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Critical
> Attachments: YARN-10501.002.patch, YARN-10501.003.patch,
> YARN-10501.004.patch
>
>
> When add a label to nodes without nodemanager port or use WILDCARD_PORT (0)
> port, it can't remove all label info in these nodes
> Reproduce process:
> {code:java}
> 1.yarn rmadmin -addToClusterNodeLabels "cpunode(exclusive=true)"
> 2.yarn rmadmin -replaceLabelsOnNode "server001=cpunode"
> 3.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":["server001:0","server001:45454"],"partitionInfo":{"resourceAvailable":{"memory":"510","vCores":"1","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"510"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"1"}]}}}
> 4.yarn rmadmin -replaceLabelsOnNode "server001"
> 5.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":"server001:45454","partitionInfo":{"resourceAvailable":{"memory":"0","vCores":"0","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"0"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"0"}]}}}
> {code}
> You can see after the 4 process to remove nodemanager labels, the label info
> is still in the node info.
> {code:java}
> 641 case REPLACE:
> 642 replaceNodeForLabels(nodeId, host.labels, labels);
> 643 replaceLabelsForNode(nodeId, host.labels, labels);
> 644 host.labels.clear();
> 645 host.labels.addAll(labels);
> 646 for (Node node : host.nms.values()) {
> 647 replaceNodeForLabels(node.nodeId, node.labels, labels);
> 649 node.labels = null;
> 650 }
> 651 break;{code}
> The cause is in 647 line, when add labels to node without port, the 0 port
> and the real nm port with be both add to node info, and when remove labels,
> the parameter node.labels in 647 line is null, so it will not remove the old
> label.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10501) Can't remove all node labels after add node label without nodemanager port
[
https://issues.apache.org/jira/browse/YARN-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
caozhiqiang updated YARN-10501:
---
Attachment: YARN-10501.004.patch
> Can't remove all node labels after add node label without nodemanager port
> --
>
> Key: YARN-10501
> URL: https://issues.apache.org/jira/browse/YARN-10501
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Critical
> Attachments: YARN-10501.002.patch, YARN-10501.003.patch,
> YARN-10501.004.patch
>
>
> When add a label to nodes without nodemanager port or use WILDCARD_PORT (0)
> port, it can't remove all label info in these nodes
> Reproduce process:
> {code:java}
> 1.yarn rmadmin -addToClusterNodeLabels "cpunode(exclusive=true)"
> 2.yarn rmadmin -replaceLabelsOnNode "server001=cpunode"
> 3.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":["server001:0","server001:45454"],"partitionInfo":{"resourceAvailable":{"memory":"510","vCores":"1","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"510"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"1"}]}}}
> 4.yarn rmadmin -replaceLabelsOnNode "server001"
> 5.curl http://RM_IP:8088/ws/v1/cluster/label-mappings
> {"labelsToNodes":{"entry":{"key":{"name":"cpunode","exclusivity":"true"},"value":{"nodes":"server001:45454","partitionInfo":{"resourceAvailable":{"memory":"0","vCores":"0","resourceInformations":{"resourceInformation":[{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"memory-mb","resourceType":"COUNTABLE","units":"Mi","value":"0"},{"attributes":null,"maximumAllocation":"9223372036854775807","minimumAllocation":"0","name":"vcores","resourceType":"COUNTABLE","units":"","value":"0"}]}}}
> {code}
> You can see after the 4 process to remove nodemanager labels, the label info
> is still in the node info.
> {code:java}
> 641 case REPLACE:
> 642 replaceNodeForLabels(nodeId, host.labels, labels);
> 643 replaceLabelsForNode(nodeId, host.labels, labels);
> 644 host.labels.clear();
> 645 host.labels.addAll(labels);
> 646 for (Node node : host.nms.values()) {
> 647 replaceNodeForLabels(node.nodeId, node.labels, labels);
> 649 node.labels = null;
> 650 }
> 651 break;{code}
> The cause is in 647 line, when add labels to node without port, the 0 port
> and the real nm port with be both add to node info, and when remove labels,
> the parameter node.labels in 647 line is null, so it will not remove the old
> label.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287107#comment-17287107
]
Hadoop QA commented on YARN-10623:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
35s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 29s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
11s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
34s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 35s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Comment Edited] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287065#comment-17287065 ] Peter Bacsko edited comment on YARN-10636 at 2/19/21, 1:00 PM: --- Thanks [~shuzirra] for the patch and [~gandras]/[~zhuqi]/[~bteke] for the review. Since we're in a big hurry, I already committed v2 w/o Jenkins results. Once Jenkins posts here, I'll check it and (hopefully) close the ticket. EDIT: there will be no Jenkins, because the changeset is already in :D anyway v1-v2 difference is minimal. was (Author: pbacsko): Thanks [~shuzirra] for the patch and [~gandras]/[~zhuqi]/[~bteke] for the review. Since we're in a big hurry, I already committed v2 w/o Jenkins results. Once Jenkins posts here, I'll check it and (hopefully) close the ticket. > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch, YARN-10636.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287065#comment-17287065 ] Peter Bacsko commented on YARN-10636: - Thanks [~shuzirra] for the patch and [~gandras]/[~zhuqi]/[~bteke] for the review. Since we're in a big hurry, I already committed v2 w/o Jenkins results. Once Jenkins posts here, I'll check it and (hopefully) close the ticket. > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch, YARN-10636.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[
https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287063#comment-17287063
]
Hadoop QA commented on YARN-10636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10636 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10636 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020700/YARN-10636.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/642/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> CS Auto Queue creation should reject submissions with empty path parts
> --
>
> Key: YARN-10636
> URL: https://issues.apache.org/jira/browse/YARN-10636
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10636.001.patch, YARN-10636.002.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10636: -- Attachment: YARN-10636.002.patch > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch, YARN-10636.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287056#comment-17287056 ] Gergely Pollak commented on YARN-10636: --- Thank you for the feedbacks [~zhuqi] [~gandras] [~bteke] much appreciated! [~bteke] [~gandras] I'd suggest to create a followup Jira for test refinements. Fixed the checkstyle issue and the redundant null check in patchset 2. Test failures seem to be unrelated, manually ran them, all passed. > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch, YARN-10636.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts
[ https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287005#comment-17287005 ] Peter Bacsko commented on YARN-10635: - +1 Thanks [~shuzirra] for the patch and [~bteke], [~shuzirra], [~zhuqi] for the review. Patch has been committed to trunk. > CSMapping rule can return paths with empty parts > > > Key: YARN-10635 > URL: https://issues.apache.org/jira/browse/YARN-10635 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10635.001.patch, YARN-10635.002.patch, > YARN-10635.003.patch > > > When a variable to be substituted evaluates to empty string, we might result > with paths where one of the parts is empty, these paths are obviously > problematic, but sometimes (when the path includes a dynamicParent) we accept > them as valid paths instead of getting the fallback action of the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[ https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287001#comment-17287001 ] Qi Zhu commented on YARN-10623: --- Updated patch to fix test, and checkstyle. > Capacity scheduler should support refresh queue automatically by a thread > policy. > - > > Key: YARN-10623 > URL: https://issues.apache.org/jira/browse/YARN-10623 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10623.001.patch, YARN-10623.002.patch, > YARN-10623.003.patch > > > In fair scheduler, it is supported that refresh queue related conf > automatically by a thread to reload, but in capacity scheduler we only > support to refresh queue related changes by refreshQueues, it is needed for > our cluster to realize queue manage. > cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10414) Improve logging in CS placement rules to help debuggability
[ https://issues.apache.org/jira/browse/YARN-10414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287000#comment-17287000 ] Gergely Pollak commented on YARN-10414: --- A message improvement suggestion was provided in: https://issues.apache.org/jira/browse/YARN-10635 > Improve logging in CS placement rules to help debuggability > --- > > Key: YARN-10414 > URL: https://issues.apache.org/jira/browse/YARN-10414 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > It should be always clear which MappingRule determined the placement (or > rejection) of the application, in the case of debug logging the whys should > also be more visible. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[ https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10623: -- Attachment: YARN-10623.003.patch > Capacity scheduler should support refresh queue automatically by a thread > policy. > - > > Key: YARN-10623 > URL: https://issues.apache.org/jira/browse/YARN-10623 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10623.001.patch, YARN-10623.002.patch, > YARN-10623.003.patch > > > In fair scheduler, it is supported that refresh queue related conf > automatically by a thread to reload, but in capacity scheduler we only > support to refresh queue related changes by refreshQueues, it is needed for > our cluster to realize queue manage. > cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9615) Add dispatcher metrics to RM
[
https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286976#comment-17286976
]
Hadoop QA commented on YARN-9615:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
13s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 5s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
2s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
30s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
30s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
11s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 33s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/640/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new +
53 unchanged - 0 fixed = 54 total (was 53) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace
[jira] [Updated] (YARN-10638) Add fair call queue support to event processing queue.
[ https://issues.apache.org/jira/browse/YARN-10638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10638: -- Description: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair queue client RPC based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom's affection to yarn scheduler performance. cc [~epayne] [~jhung] [~wangda] [~ztang] [~bibinchundatt] [~hcarrot] [~gandras] [~bteke] [~ebadger] If you have any advice for this proposal. was: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair queue client RPC based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. cc [~epayne] [~jhung] [~wangda] [~ztang] [~bibinchundatt] [~hcarrot] [~gandras] [~bteke] [~ebadger] If you have any advice for this proposal. > Add fair call queue support to event processing queue. > -- > > Key: YARN-10638 > URL: https://issues.apache.org/jira/browse/YARN-10638 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > HADOOP-15016 has support the fair call queue. > In HDFS-14403, hdfs support fair queue client RPC based user, i think the > fair call also can be used to YARN event queue: > When event boom, the fair can let other normal event go, which can > improvement the event boom's affection to yarn scheduler performance. > > cc [~epayne] [~jhung] [~wangda] [~ztang] [~bibinchundatt] [~hcarrot] > [~gandras] > [~bteke] [~ebadger] > If you have any advice for this proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286971#comment-17286971
]
Hadoop QA commented on YARN-10623:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
48s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 19s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
0s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
13s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
12s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/639/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 8 new +
184 unchanged - 0 fixed = 192 total (was 184) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color}
[jira] [Updated] (YARN-10638) Add fair call queue support to event processing queue.
[ https://issues.apache.org/jira/browse/YARN-10638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10638: -- Description: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair queue client RPC based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. cc [~epayne] [~jhung] [~wangda] [~ztang] [~bibinchundatt] [~hcarrot] [~gandras] [~bteke] [~ebadger] If you have any advice for this proposal. was: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair queue client RPC based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. cc [~epayne] [~wangda] [~ztang] [~bibinchundatt] If you have any advice for this proposal. > Add fair call queue support to event processing queue. > -- > > Key: YARN-10638 > URL: https://issues.apache.org/jira/browse/YARN-10638 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > HADOOP-15016 has support the fair call queue. > In HDFS-14403, hdfs support fair queue client RPC based user, i think the > fair call also can be used to YARN event queue: > When event boom, the fair can let other normal event go, which can > improvement the event boom will affect the yarn scheduler performance. > > cc [~epayne] [~jhung] [~wangda] [~ztang] [~bibinchundatt] [~hcarrot] > [~gandras] > [~bteke] [~ebadger] > If you have any advice for this proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10638) Add fair call queue support to event processing queue.
[ https://issues.apache.org/jira/browse/YARN-10638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10638: -- Description: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair queue client RPC based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. cc [~epayne] [~wangda] [~ztang] [~bibinchundatt] If you have any advice for this proposal. was: HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. > Add fair call queue support to event processing queue. > -- > > Key: YARN-10638 > URL: https://issues.apache.org/jira/browse/YARN-10638 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > HADOOP-15016 has support the fair call queue. > In HDFS-14403, hdfs support fair queue client RPC based user, i think the > fair call also can be used to YARN event queue: > When event boom, the fair can let other normal event go, which can > improvement the event boom will affect the yarn scheduler performance. > > cc [~epayne] [~wangda] [~ztang] [~bibinchundatt] > If you have any advice for this proposal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10638) Add fair call queue support to event processing queue.
Qi Zhu created YARN-10638: - Summary: Add fair call queue support to event processing queue. Key: YARN-10638 URL: https://issues.apache.org/jira/browse/YARN-10638 Project: Hadoop YARN Issue Type: New Feature Reporter: Qi Zhu Assignee: Qi Zhu HADOOP-15016 has support the fair call queue. In HDFS-14403, hdfs support fair based user, i think the fair call also can be used to YARN event queue: When event boom, the fair can let other normal event go, which can improvement the event boom will affect the yarn scheduler performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9927) RM multi-thread event processing mechanism
[ https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu reassigned YARN-9927: Assignee: Qi Zhu (was: Bilwa S T) > RM multi-thread event processing mechanism > -- > > Key: YARN-9927 > URL: https://issues.apache.org/jira/browse/YARN-9927 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.0.0, 2.9.2 >Reporter: hcarrot >Assignee: Qi Zhu >Priority: Major > Attachments: RM multi-thread event processing mechanism.pdf, > YARN-9927.001.patch > > > Recently, we have observed serious event blocking in RM event dispatcher > queue. After analysis of RM event monitoring data and RM event processing > logic, we found that > 1) environment: a cluster with thousands of nodes > 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler > 3) Meanwhile, RM event processing is in a single-thread mode, and It results > in the low headroom of RM event scheduler, thus performance of RM. > So we proposed a RM multi-thread event processing mechanism to improve RM > performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286949#comment-17286949 ] Benjamin Teke commented on YARN-10636: -- Thanks [~shuzirra]. I too think the new test case should be introduced in TestCapacitySchedulerNewQueueAutoCreation. If you fix the checkstyle issue could you move it to that class? Thanks! > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts
[ https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286948#comment-17286948 ] Qi Zhu commented on YARN-10635: --- Thanks [~shuzirra] for your contribution, the patch LGTM. > CSMapping rule can return paths with empty parts > > > Key: YARN-10635 > URL: https://issues.apache.org/jira/browse/YARN-10635 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10635.001.patch, YARN-10635.002.patch, > YARN-10635.003.patch > > > When a variable to be substituted evaluates to empty string, we might result > with paths where one of the parts is empty, these paths are obviously > problematic, but sometimes (when the path includes a dynamicParent) we accept > them as valid paths instead of getting the fallback action of the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts
[ https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286947#comment-17286947 ] Benjamin Teke commented on YARN-10635: -- [~shuzirra] Thanks for the patch. I agree with [~gandras]. LGTM with that minor thing. > CSMapping rule can return paths with empty parts > > > Key: YARN-10635 > URL: https://issues.apache.org/jira/browse/YARN-10635 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10635.001.patch, YARN-10635.002.patch, > YARN-10635.003.patch > > > When a variable to be substituted evaluates to empty string, we might result > with paths where one of the parts is empty, these paths are obviously > problematic, but sometimes (when the path includes a dynamicParent) we accept > them as valid paths instead of getting the fallback action of the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10637) We should support fs to cs support for auto refresh queues when conf changed, after YARN-10623 finished.
[ https://issues.apache.org/jira/browse/YARN-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10637: -- Description: cc [~pbacsko] [~gandras] [~bteke] We should also fill this, when YARN-10623 finished. was: cc [~pbacsko] [~gandras] [~bteke] We should also fill this, when > We should support fs to cs support for auto refresh queues when conf changed, > after YARN-10623 finished. > > > Key: YARN-10637 > URL: https://issues.apache.org/jira/browse/YARN-10637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Priority: Major > > cc [~pbacsko] [~gandras] [~bteke] > We should also fill this, when YARN-10623 finished. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10637) We should support fs to cs support for auto refresh queues when conf changed, after YARN-10623 finished.
[ https://issues.apache.org/jira/browse/YARN-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated YARN-10637: -- Description: cc [~pbacsko] [~gandras] [~bteke] We should also fill this, when > We should support fs to cs support for auto refresh queues when conf changed, > after YARN-10623 finished. > > > Key: YARN-10637 > URL: https://issues.apache.org/jira/browse/YARN-10637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Priority: Major > > cc [~pbacsko] [~gandras] [~bteke] > We should also fill this, when -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286943#comment-17286943 ] Andras Gyori commented on YARN-10636: - Thank you [~shuzirra]. The patch LGTM, I have only some minor addition (ignore this, if no new revision is needed). * the TestCapacitySchedulerNewQueueAutoCreation has a lot of utility to ease the testing of this behaviour (because you are checking this scenario in case of AQC V2). * TestCapacitySchedulerAutoQueueCreation:581 is underlined, because it seems to be a redundant null check. > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts
[ https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286924#comment-17286924 ] Andras Gyori commented on YARN-10635: - Thank you [~shuzirra] for the fix. I see no faults in this patch, I can only think of one very minor suggestion (if there will be no more revisions, just ignore this). From the user's point of view, maybe a more specific error message would be appropriate when an empty path is encountered instead of Invalid path (because we know exactly why it has failed). All in all, LGTM +1. > CSMapping rule can return paths with empty parts > > > Key: YARN-10635 > URL: https://issues.apache.org/jira/browse/YARN-10635 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10635.001.patch, YARN-10635.002.patch, > YARN-10635.003.patch > > > When a variable to be substituted evaluates to empty string, we might result > with paths where one of the parts is empty, these paths are obviously > problematic, but sometimes (when the path includes a dynamicParent) we accept > them as valid paths instead of getting the fallback action of the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
