Re: Regarding containers not launching

[Eric Payne]

Nishchay Malhotra, what scheduler are you using? Also, what are the settings 
for each queue?



  From: Billy Watson <williamrwat...@gmail.com>
 To: nishchay malhotra <nishchay.malht...@gmail.com> 
Cc: "common-u...@hadoop.apache.org" <user@hadoop.apache.org>
 Sent: Tuesday, January 30, 2018 9:47 AM
 Subject: Re: Regarding containers not launching
   
Is your job able to use more containers, I.e. does your job have tasks waiting 
or are all tasks in progress?
William Watson


On Tue, Jan 30, 2018 at 1:56 AM, nishchay malhotra 
<nishchay.malht...@gmail.com> wrote:

What should I be looking for if my 24-node cluster in not launching enough 
containers?
only 40/288 cores are used and 87GB/700GB is memory is used.
Yarn.nodemanager memory/core conf look good. And so do container memory/core 
conf.

Thanks 
Nishchay Malhotra




   

Re: Regarding containers not launching

[Billy Watson]

Also, is there anything interesting in the yarn scheduler logs? Something
about scheduling being skipped?




On Wed, Jan 31, 2018 at 05:16 Billy Watson  wrote:

> Ok, and your container settings?
>
> On Wed, Jan 31, 2018 at 02:38 nishchay malhotra <
> nishchay.malht...@gmail.com> wrote:
>
>> yes my job has about 160,000 maps and my cluster not getting fully
>> utilized around 6000 maps ran for 2 hrs and then I killed the job. At any
>> point of time only 40 containers are running thats just 11% of my cluster
>> capacity.
>>
>> {
>> "classification": "mapred-site",
>> "properties": {
>>   "mapreduce.job.reduce.slowstart.completedmaps":"1",
>>   "mapreduce.reduce.memory.mb": "3072",
>>   "mapreduce.map.memory.mb": "2208",
>>   "mapreduce.map.java.opts":"-Xmx1800m",
>>   "mapreduce.map.cpu.vcores":"1"
>> }
>>   },
>>   {
>>   "classification": "yarn-site",
>>   "properties": {
>> "yarn.scheduler.minimum-allocation-mb": "32”,
>> “yarn.scheduler.maximum-allocation-mb”:”253952”,
>> “yarn.scheduler.maximum-allocation-vcores: “128”
>>
>> "yarn.nodemanager.vmem-pmem-ratio":"3",
>> "yarn.nodemanager.vmem-check-enabled":"true",
>>  yarn.nodemanager.resource.cpu-vcores" ; "16”,
>>  yarn.nodemanager.resource.memory-mb: “23040"
>>   }
>>
>> Each node: capacity
>> Disk-space=100gb
>> memory=28gb
>> processors: 8
>>
>>
>> --
> William Watson
>
> --
William Watson

Re: Regarding containers not launching

[Billy Watson]

Ok, and your container settings?

On Wed, Jan 31, 2018 at 02:38 nishchay malhotra 
wrote:

> yes my job has about 160,000 maps and my cluster not getting fully
> utilized around 6000 maps ran for 2 hrs and then I killed the job. At any
> point of time only 40 containers are running thats just 11% of my cluster
> capacity.
>
> {
> "classification": "mapred-site",
> "properties": {
>   "mapreduce.job.reduce.slowstart.completedmaps":"1",
>   "mapreduce.reduce.memory.mb": "3072",
>   "mapreduce.map.memory.mb": "2208",
>   "mapreduce.map.java.opts":"-Xmx1800m",
>   "mapreduce.map.cpu.vcores":"1"
> }
>   },
>   {
>   "classification": "yarn-site",
>   "properties": {
> "yarn.scheduler.minimum-allocation-mb": "32”,
> “yarn.scheduler.maximum-allocation-mb”:”253952”,
> “yarn.scheduler.maximum-allocation-vcores: “128”
>
> "yarn.nodemanager.vmem-pmem-ratio":"3",
> "yarn.nodemanager.vmem-check-enabled":"true",
>  yarn.nodemanager.resource.cpu-vcores" ; "16”,
>  yarn.nodemanager.resource.memory-mb: “23040"
>   }
>
> Each node: capacity
> Disk-space=100gb
> memory=28gb
> processors: 8
>
>
> --
William Watson

Re: Regarding containers not launching

[nishchay malhotra]

yes my job has about 160,000 maps and my cluster not getting fully utilized
around 6000 maps ran for 2 hrs and then I killed the job. At any point of
time only 40 containers are running thats just 11% of my cluster capacity.

{
"classification": "mapred-site",
"properties": {
  "mapreduce.job.reduce.slowstart.completedmaps":"1",
  "mapreduce.reduce.memory.mb": "3072",
  "mapreduce.map.memory.mb": "2208",
  "mapreduce.map.java.opts":"-Xmx1800m",
  "mapreduce.map.cpu.vcores":"1"
}
  },
  {
  "classification": "yarn-site",
  "properties": {
"yarn.scheduler.minimum-allocation-mb": "32”,
“yarn.scheduler.maximum-allocation-mb”:”253952”,
“yarn.scheduler.maximum-allocation-vcores: “128”

"yarn.nodemanager.vmem-pmem-ratio":"3",
"yarn.nodemanager.vmem-check-enabled":"true",
 yarn.nodemanager.resource.cpu-vcores" ; "16”,
 yarn.nodemanager.resource.memory-mb: “23040"
  }

Each node: capacity
Disk-space=100gb
memory=28gb
processors: 8

Re: Regarding containers not launching

[Billy Watson]

Is your job able to use more containers, I.e. does your job have tasks
waiting or are all tasks in progress?

William Watson


On Tue, Jan 30, 2018 at 1:56 AM, nishchay malhotra <
nishchay.malht...@gmail.com> wrote:

> What should I be looking for if my 24-node cluster in not launching enough
> containers?
> only 40/288 cores are used and 87GB/700GB is memory is used.
> Yarn.nodemanager memory/core conf look good. And so do container
> memory/core conf.
>
> Thanks
> Nishchay Malhotra
>