[jira] [Commented] (YARN-10360) Support Multi Node Placement in SingleConstraintAppPlacementAllocator

2020-08-23 Thread Prabhu Joseph (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182958#comment-17182958
 ] 

Prabhu Joseph commented on YARN-10360:
--

Thanks [~sunilg] for the review.

Testcase failures are existing ones and tracked by YARN-9333 and YARN-9587.

> Support Multi Node Placement in SingleConstraintAppPlacementAllocator
> -
>
> Key: YARN-10360
> URL: https://issues.apache.org/jira/browse/YARN-10360
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler, multi-node-placement
>Affects Versions: 3.4.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10360-001.patch, YARN-10360-002.patch
>
>
> Currently, placement constraints are not supported when Multi Node Placement 
> is enabled. This Jira is to add Support for Multi Node Placement in 
> SingleConstraintAppPlacementAllocator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10398) Every NM will try to upload Jar/Archives/Files/Resources to Yarn Shared Cache Manager Like DDOS

2020-08-23 Thread zhenzhao wang (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182949#comment-17182949
 ] 

zhenzhao wang commented on YARN-10398:
--

[~jiwq] I double checked and confirmed the PR is the fix for the problem. The 
reason why non-application master try to upload is because the clear code 
didn't work. The code and bug are in YARN. MR uses yarn shared cache. I'm not 
sure we should move it MR project.  Thanks.

> Every NM will try to upload Jar/Archives/Files/Resources to Yarn Shared Cache 
> Manager Like DDOS
> ---
>
> Key: YARN-10398
> URL: https://issues.apache.org/jira/browse/YARN-10398
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 3.0.2, 3.2.0, 3.1.1, 
> 2.9.2, 3.0.3, 3.0.4, 3.1.2, 3.3.0, 3.2.1, 2.9.3, 3.1.3, 3.2.2, 3.1.4, 3.4.0, 
> 3.3.1, 3.1.5
>Reporter: zhenzhao wang
>Assignee: zhenzhao wang
>Priority: Major
>
> The design of yarn shared cache manager is only to allow application master 
> should upload the jar/files/resource. However, there was a bug in the code 
> since 2.9.0. Every node manager that take the job task will try to upload the 
> jar/resources. Let's say one job have 5000 tasks. Then there will be up to 
> 5000 NMs try to upload the jar. This is like DDOS and create a snowball 
> effect. It will end up with inavailability of yarn shared cache manager. It 
> wil cause time out in localization and lead to job failure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-10404) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Gary reopened YARN-10404:
-

> ResourseManager use HA mode both in Standby
> ---
>
> Key: YARN-10404
> URL: https://issues.apache.org/jira/browse/YARN-10404
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.1.1
>Reporter: Jin Gary
>Priority: Major
> Attachments: image-2020-08-24-11-07-50-899.png
>
>
> our yarn use capacity scheduler, config is 
> {code:java}
> yarn.scheduler.capacity.maximum-am-resource-percent=0.2
> yarn.scheduler.capacity.maximum-applications=1
> yarn.scheduler.capacity.node-locality-delay=20
> yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.acl_submit_applications=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=1
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.default.maximum-applications=10
> yarn.scheduler.capacity.root.default.maximum-capacity=30
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.priority=0
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=10
> yarn.scheduler.capacity.root.developer.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.capacity=7
> yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_data.capacity=50
> yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_data.priority=20
> yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
> yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.developer.maximum-capacity=40
> yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
> yarn.scheduler.capacity.root.developer.priority=20
> yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
> yarn.scheduler.capacity.root.developer.state=RUNNING
> yarn.scheduler.capacity.root.developer.user-limit-factor=3
> yarn.scheduler.capacity.root.olap.acl_administer_queue=*
> yarn.scheduler.capacity.root.olap.acl_submit_applications=*
> yarn.scheduler.capacity.root.olap.capacity=5
> yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.olap.maximum-capacity=30
> yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.olap.ordering-policy=fifo
> yarn.scheduler.capacity.root.olap.priority=50
> yarn.scheduler.capacity.root.olap.state=RUNNING
>

[jira] [Resolved] (YARN-10405) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Gary resolved YARN-10405.
-
Resolution: Duplicate

> ResourseManager use HA mode both in Standby
> ---
>
> Key: YARN-10405
> URL: https://issues.apache.org/jira/browse/YARN-10405
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Jin Gary
>Priority: Major
>
> our yarn use capacity scheduler, config is 
> {code:java}
> yarn.scheduler.capacity.maximum-am-resource-percent=0.2
> yarn.scheduler.capacity.maximum-applications=1
> yarn.scheduler.capacity.node-locality-delay=20
> yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.acl_submit_applications=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=1
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.default.maximum-applications=10
> yarn.scheduler.capacity.root.default.maximum-capacity=30
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.priority=0
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=10
> yarn.scheduler.capacity.root.developer.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.capacity=7
> yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_data.capacity=50
> yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_data.priority=20
> yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
> yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.developer.maximum-capacity=40
> yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
> yarn.scheduler.capacity.root.developer.priority=20
> yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
> yarn.scheduler.capacity.root.developer.state=RUNNING
> yarn.scheduler.capacity.root.developer.user-limit-factor=3
> yarn.scheduler.capacity.root.olap.acl_administer_queue=*
> yarn.scheduler.capacity.root.olap.acl_submit_applications=*
> yarn.scheduler.capacity.root.olap.capacity=5
> yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.olap.maximum-capacity=30
> yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.olap.ordering-policy=fifo
> yarn.scheduler.capacity.root.olap.priority=50
> yarn.scheduler.capacity.root.olap.state=RUNNING
> yarn.scheduler.capacity.root.olap.user-limit-factor=3
>

[jira] [Created] (YARN-10405) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 
Jin Gary created YARN-10405:
---

 Summary: ResourseManager use HA mode both in Standby
 Key: YARN-10405
 URL: https://issues.apache.org/jira/browse/YARN-10405
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Jin Gary


our yarn use capacity scheduler, config is 
{code:java}
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=1
yarn.scheduler.capacity.node-locality-delay=20
yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=1
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.default.maximum-applications=10
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=10
yarn.scheduler.capacity.root.developer.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.capacity=7
yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_data.capacity=50
yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_data.priority=20
yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.developer.maximum-capacity=40
yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.developer.priority=20
yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
yarn.scheduler.capacity.root.developer.state=RUNNING
yarn.scheduler.capacity.root.developer.user-limit-factor=3
yarn.scheduler.capacity.root.olap.acl_administer_queue=*
yarn.scheduler.capacity.root.olap.acl_submit_applications=*
yarn.scheduler.capacity.root.olap.capacity=5
yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.olap.maximum-capacity=30
yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.olap.ordering-policy=fifo
yarn.scheduler.capacity.root.olap.priority=50
yarn.scheduler.capacity.root.olap.state=RUNNING
yarn.scheduler.capacity.root.olap.user-limit-factor=3
yarn.scheduler.capacity.root.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.pipeline.acl_administer_queue=*
yarn.scheduler.capacity.root.pipeline.acl_submit_applications=*
yarn.scheduler.capacity.root.pipeline.capacity=30
yarn.scheduler.capacity.root.pipeline.maximum-allocation-mb=21504
yarn.scheduler.capacity.root.pipeline.maximum-capacity=30

[jira] [Resolved] (YARN-10404) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Gary resolved YARN-10404.
-
Resolution: Duplicate

> ResourseManager use HA mode both in Standby
> ---
>
> Key: YARN-10404
> URL: https://issues.apache.org/jira/browse/YARN-10404
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.1.1
>Reporter: Jin Gary
>Priority: Major
> Attachments: image-2020-08-24-11-07-50-899.png
>
>
> our yarn use capacity scheduler, config is 
> {code:java}
> yarn.scheduler.capacity.maximum-am-resource-percent=0.2
> yarn.scheduler.capacity.maximum-applications=1
> yarn.scheduler.capacity.node-locality-delay=20
> yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
> yarn.scheduler.capacity.queue-mappings-override.enable=true
> yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
> yarn.scheduler.capacity.root.acl_administer_queue=*
> yarn.scheduler.capacity.root.acl_submit_applications=*
> yarn.scheduler.capacity.root.capacity=100
> yarn.scheduler.capacity.root.default.acl_submit_applications=*
> yarn.scheduler.capacity.root.default.capacity=1
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.default.maximum-applications=10
> yarn.scheduler.capacity.root.default.maximum-capacity=30
> yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.default.ordering-policy=fair
> yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.default.priority=0
> yarn.scheduler.capacity.root.default.state=RUNNING
> yarn.scheduler.capacity.root.default.user-limit-factor=10
> yarn.scheduler.capacity.root.developer.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.capacity=7
> yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_data.capacity=50
> yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_data.priority=20
> yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
> yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
> yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
> yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
> yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
> yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
> yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
> yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
> yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
> yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.developer.maximum-capacity=40
> yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
> yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
> yarn.scheduler.capacity.root.developer.priority=20
> yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
> yarn.scheduler.capacity.root.developer.state=RUNNING
> yarn.scheduler.capacity.root.developer.user-limit-factor=3
> yarn.scheduler.capacity.root.olap.acl_administer_queue=*
> yarn.scheduler.capacity.root.olap.acl_submit_applications=*
> yarn.scheduler.capacity.root.olap.capacity=5
> yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
> yarn.scheduler.capacity.root.olap.maximum-capacity=30
> yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
> yarn.scheduler.capacity.root.olap.ordering-policy=fifo
> yarn.scheduler.capacity.root.olap.priority=50
> yarn.scheduler.capacity.root.olap.state=RUNNING
>

[jira] [Updated] (YARN-10404) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Gary updated YARN-10404:

Description: 
our yarn use capacity scheduler, config is 
{code:java}
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=1
yarn.scheduler.capacity.node-locality-delay=20
yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=1
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.default.maximum-applications=10
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=10
yarn.scheduler.capacity.root.developer.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.capacity=7
yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_data.capacity=50
yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_data.priority=20
yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.developer.maximum-capacity=40
yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.developer.priority=20
yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
yarn.scheduler.capacity.root.developer.state=RUNNING
yarn.scheduler.capacity.root.developer.user-limit-factor=3
yarn.scheduler.capacity.root.olap.acl_administer_queue=*
yarn.scheduler.capacity.root.olap.acl_submit_applications=*
yarn.scheduler.capacity.root.olap.capacity=5
yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.olap.maximum-capacity=30
yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.olap.ordering-policy=fifo
yarn.scheduler.capacity.root.olap.priority=50
yarn.scheduler.capacity.root.olap.state=RUNNING
yarn.scheduler.capacity.root.olap.user-limit-factor=3
yarn.scheduler.capacity.root.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.pipeline.acl_administer_queue=*
yarn.scheduler.capacity.root.pipeline.acl_submit_applications=*
yarn.scheduler.capacity.root.pipeline.capacity=30
yarn.scheduler.capacity.root.pipeline.maximum-allocation-mb=21504
yarn.scheduler.capacity.root.pipeline.maximum-capacity=30
yarn.scheduler.capacity.root.pipeline.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.pipeline.ordering-policy=fifo
yarn.scheduler.capacity.root.pipeline.priority=60

[jira] [Updated] (YARN-10404) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 


 [ 
https://issues.apache.org/jira/browse/YARN-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Gary updated YARN-10404:

Description: 
our yarn use capacity scheduler, config is 
{code:java}
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=1
yarn.scheduler.capacity.node-locality-delay=20
yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=1
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.default.maximum-applications=10
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=10
yarn.scheduler.capacity.root.developer.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.capacity=7
yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_data.capacity=50
yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_data.priority=20
yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.developer.maximum-capacity=40
yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.developer.priority=20
yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
yarn.scheduler.capacity.root.developer.state=RUNNING
yarn.scheduler.capacity.root.developer.user-limit-factor=3
yarn.scheduler.capacity.root.olap.acl_administer_queue=*
yarn.scheduler.capacity.root.olap.acl_submit_applications=*
yarn.scheduler.capacity.root.olap.capacity=5
yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.olap.maximum-capacity=30
yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.olap.ordering-policy=fifo
yarn.scheduler.capacity.root.olap.priority=50
yarn.scheduler.capacity.root.olap.state=RUNNING
yarn.scheduler.capacity.root.olap.user-limit-factor=3
yarn.scheduler.capacity.root.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.pipeline.acl_administer_queue=*
yarn.scheduler.capacity.root.pipeline.acl_submit_applications=*
yarn.scheduler.capacity.root.pipeline.capacity=30
yarn.scheduler.capacity.root.pipeline.maximum-allocation-mb=21504
yarn.scheduler.capacity.root.pipeline.maximum-capacity=30
yarn.scheduler.capacity.root.pipeline.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.pipeline.ordering-policy=fifo
yarn.scheduler.capacity.root.pipeline.priority=60

[jira] [Created] (YARN-10404) ResourseManager use HA mode both in Standby

2020-08-23 Thread Jin Gary (Jira) 
Jin Gary created YARN-10404:
---

 Summary: ResourseManager use HA mode both in Standby
 Key: YARN-10404
 URL: https://issues.apache.org/jira/browse/YARN-10404
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 3.1.1
Reporter: Jin Gary
 Attachments: image-2020-08-24-11-07-50-899.png

our yarn use capacity scheduler, config is 
{code:java}
//代码占位符 yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=1
yarn.scheduler.capacity.node-locality-delay=20
yarn.scheduler.capacity.queue-mappings=u:username:queuename,...
yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=1
yarn.scheduler.capacity.root.default.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.default.maximum-applications=10
yarn.scheduler.capacity.root.default.maximum-capacity=30
yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.default.ordering-policy=fair
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.default.priority=0
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=10
yarn.scheduler.capacity.root.developer.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.capacity=7
yarn.scheduler.capacity.root.developer.developer_data.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_data.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_data.capacity=50
yarn.scheduler.capacity.root.developer.developer_data.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_data.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_data.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_data.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_data.priority=20
yarn.scheduler.capacity.root.developer.developer_data.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_data.user-limit-factor=5
yarn.scheduler.capacity.root.developer.developer_recsys.acl_administer_queue=*
yarn.scheduler.capacity.root.developer.developer_recsys.acl_submit_applications=*
yarn.scheduler.capacity.root.developer.developer_recsys.capacity=50
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-am-resource-percent=0.8
yarn.scheduler.capacity.root.developer.developer_recsys.maximum-capacity=70
yarn.scheduler.capacity.root.developer.developer_recsys.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy=fair
yarn.scheduler.capacity.root.developer.developer_recsys.ordering-policy.fair.enable-size-based-weight=false
yarn.scheduler.capacity.root.developer.developer_recsys.priority=20
yarn.scheduler.capacity.root.developer.developer_recsys.state=RUNNING
yarn.scheduler.capacity.root.developer.developer_recsys.user-limit-factor=5
yarn.scheduler.capacity.root.developer.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.developer.maximum-capacity=40
yarn.scheduler.capacity.root.developer.minimum-user-limit-percent=30
yarn.scheduler.capacity.root.developer.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.developer.priority=20
yarn.scheduler.capacity.root.developer.queues=developer_data,developer_recsys
yarn.scheduler.capacity.root.developer.state=RUNNING
yarn.scheduler.capacity.root.developer.user-limit-factor=3
yarn.scheduler.capacity.root.olap.acl_administer_queue=*
yarn.scheduler.capacity.root.olap.acl_submit_applications=*
yarn.scheduler.capacity.root.olap.capacity=5
yarn.scheduler.capacity.root.olap.maximum-am-resource-percent=1
yarn.scheduler.capacity.root.olap.maximum-capacity=30
yarn.scheduler.capacity.root.olap.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.olap.ordering-policy=fifo
yarn.scheduler.capacity.root.olap.priority=50
yarn.scheduler.capacity.root.olap.state=RUNNING
yarn.scheduler.capacity.root.olap.user-limit-factor=3
yarn.scheduler.capacity.root.ordering-policy=priority-utilization
yarn.scheduler.capacity.root.pipeline.acl_administer_queue=*
yarn.scheduler.capacity.root.pipeline.acl_submit_applications=*
yarn.scheduler.capacity.root.pipeline.capacity=30
yarn.scheduler.capacity.root.pipeline.maximum-allocation-mb=21504