Re: Capacity scheduler puts all containers on one box

For a work-around, if you turn on DRF / multi-resource scheduling, you could use vcore capacities to limit the number of containers per node? On Fri, Mar 21, 2014 at 11:35 PM, Chris Riccomini wrote: > Hey Guys, > > @Vinod: We aren't overriding the default, so we must be using -1 as the > setting.

Re: Capacity scheduler puts all containers on one box

Hey Guys, @Vinod: We aren't overriding the default, so we must be using -1 as the setting. @Sandy: We aren't specifying any racks/hosts when sending the resource requests. +1 regarding introducing a similar limit in capacity scheduler. Any recommended work-arounds in the mean time? Our utilizati

Re: Capacity scheduler puts all containers on one box

yarn.scheduler.capacity.node-locality-delay will help if the app is requesting containers at particular locations, but won't help spread things out evenly otherwise. The Fair Scheduler attempts an even spread. By default, it only schedules a single container each time it considers a node. Decoup

Success: YARN-1577 PreCommit Build #3437

Jira: https://issues.apache.org/jira/browse/YARN-1577 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3437/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1577 PreCommit Build #3436

Jira: https://issues.apache.org/jira/browse/YARN-1577 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3436/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1776 PreCommit Build #3435

Jira: https://issues.apache.org/jira/browse/YARN-1776 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3435/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1776 PreCommit Build #3434

Jira: https://issues.apache.org/jira/browse/YARN-1776 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3434/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1838 PreCommit Build #3433

Jira: https://issues.apache.org/jira/browse/YARN-1838 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3433/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1776 PreCommit Build #3431

Jira: https://issues.apache.org/jira/browse/YARN-1776 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3431/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1864 PreCommit Build #3432

Jira: https://issues.apache.org/jira/browse/YARN-1864 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3432/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1577 PreCommit Build #3430

Jira: https://issues.apache.org/jira/browse/YARN-1577 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3430/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Re: Capacity scheduler puts all containers on one box

What's the value for yarn.scheduler.capacity.node-locality-delay? It is -1 by default in 2.2. We fixed the default to be a reasonable 40 (nodes in a rack) in 2.3.0 that should spread containers a bit. Thanks, +Vinod On Mar 21, 2014, at 12:48 PM, Chris Riccomini wrote: > Hey Guys, > > We're

Success: YARN-1838 PreCommit Build #3429

Jira: https://issues.apache.org/jira/browse/YARN-1838 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3429/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1775 PreCommit Build #3428

Jira: https://issues.apache.org/jira/browse/YARN-1775 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3428/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

[jira] [Created] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues

Ashwin Shankar created YARN-1864: Summary: Fair Scheduler Dynamic Hierarchical User Queues Key: YARN-1864 URL: https://issues.apache.org/jira/browse/YARN-1864 Project: Hadoop YARN Issue Type:

Failed: YARN-1838 PreCommit Build #3427

Jira: https://issues.apache.org/jira/browse/YARN-1838 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3427/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1536 PreCommit Build #3426

Jira: https://issues.apache.org/jira/browse/YARN-1536 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3426/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Capacity scheduler puts all containers on one box

Hey Guys, We're running YARN 2.2 with the capacity scheduler. Each NM is running with 40G of memory capacity. When we request a series containers with 2G of memory from a single AM, we see the RM assigning them entirely to one NM until that NM is full, and then moving on to the next, and so on.

Success: YARN-1863 PreCommit Build #3425

Jira: https://issues.apache.org/jira/browse/YARN-1863 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3425/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Failed: YARN-1536 PreCommit Build #3424

Jira: https://issues.apache.org/jira/browse/YARN-1536 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3424/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Failed: YARN-1838 PreCommit Build #3423

Jira: https://issues.apache.org/jira/browse/YARN-1838 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3423/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Success: YARN-1670 PreCommit Build #3422

Jira: https://issues.apache.org/jira/browse/YARN-1670 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3422/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

[jira] [Created] (YARN-1863) TestRMFailover fails with 'AssertionError: null'

Ted Yu created YARN-1863: Summary: TestRMFailover fails with 'AssertionError: null' Key: YARN-1863 URL: https://issues.apache.org/jira/browse/YARN-1863 Project: Hadoop YARN Issue Type: Test

Hadoop-Yarn-trunk - Build # 516 - Still Failing

See https://builds.apache.org/job/Hadoop-Yarn-trunk/516/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 16002 lines...] Results : Failed tests: TestRMFailove

Build failed in Jenkins: Hadoop-Yarn-trunk #516

See Changes: [vinodkv] YARN-1811. Fixed AMFilters in YARN to correctly accept requests from either web-app proxy or the RMs when HA is enabled. Contributed by Robert Kanter. [vinodkv] YARN-1859. Fixed WebAppProxyServlet to correctly

Success: YARN-1854 PreCommit Build #3421

Jira: https://issues.apache.org/jira/browse/YARN-1854 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3421/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat

Re: Error in running DistributedShell

Please ignore below emails, looks like Yarn do need HDFS to download applications jar on nodemanager before they can run . Thanks for the patience On 20 March 2014 18:05, Kapoor wrote: > Is it because I am not running datanode therefore ResourceLocalizationService > is unable to download the j

Success: YARN-1775 PreCommit Build #3420

Jira: https://issues.apache.org/jira/browse/YARN-1775 Build: https://builds.apache.org/job/PreCommit-YARN-Build/3420/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncat