Re: HDFS replication factor

[रविशंकर नायर]

This is solved in Hadoop 3. So stay tuned

Best,

On Feb 2, 2018 6:26 AM, "李立伟"  wrote:

> Hi:
>   It's my understanding that HDFS write  operation is not considered
> completd until all of the replicas have been successfully written.If so,
> does the replication factor affect the write latency? the mapreduce\spark
> task will be affected?
>   is there the way to set HDFS write the first replica synchronously
> and return ,the others in an asynchronous.
>   Thanks in advance.
>
>
>

Self-healing hdfs functionality

[sidharth kumar]

Hi,

I would like to have a list of issues we generally face in hdfs and how can we 
make it a self healing distributed file system. I do know there are few 
functionalities of hdfs which is self healing like under replication of blocks. 
But still there are multiple issues a hadoop administrator face day to day life 
which can be automated as self healing feature .

So I would like to request all community member to provide list of issue they 
face everyday and can be taken as a feature for self healing hdfs

Warm Regards

Sidharth Kumar | Mob: +91 8197 555 599
LinkedIn:www.linkedin.com/in/sidharthkumar2792

Re: Regarding containers not launching

[Eric Payne]

Nishchay Malhotra, what scheduler are you using? Also, what are the settings 
for each queue?



  From: Billy Watson 
 To: nishchay malhotra  
Cc: "common-u...@hadoop.apache.org" 
 Sent: Tuesday, January 30, 2018 9:47 AM
 Subject: Re: Regarding containers not launching
   
Is your job able to use more containers, I.e. does your job have tasks waiting 
or are all tasks in progress?
William Watson


On Tue, Jan 30, 2018 at 1:56 AM, nishchay malhotra 
 wrote:

What should I be looking for if my 24-node cluster in not launching enough 
containers?
only 40/288 cores are used and 87GB/700GB is memory is used.
Yarn.nodemanager memory/core conf look good. And so do container memory/core 
conf.

Thanks 
Nishchay Malhotra




   

HDFS replication factor

[李立伟]

Hi:
  It's my understanding that HDFS write  operation is not considered
completd until all of the replicas have been successfully written.If so,
does the replication factor affect the write latency? the mapreduce\spark
task will be affected?
  is there the way to set HDFS write the first replica synchronously
and return ,the others in an asynchronous.
  Thanks in advance.