Re: total vcores per node containers in yarn

2015-07-19 Thread Harsh J
That doesn't sound right, if you are indeed making use of the new feature. First off, this is not the default behaviour in the actual Apache Hadoop 2.5.0 release. Are you using a different distribution based on that release instead? Secondly, if it were using the auto-calculate feature, it'd only

HDFS how to specify the exact datanode to put data on?

2015-07-19 Thread Shiyao Ma
Hi, I'd like to put my data selectively on some datanodes. Currently I can do that by shutting down un-needed datanodes. But this is a little laborsome. Is it possible to directly specify the datanodes I'd like to put the data on when doing operations like: hdfs dfs -put / TIA. --

Re: HDFS how to specify the exact datanode to put data on?

2015-07-19 Thread Ted Yu
DFSClient#create() allows you to pass favored nodes. Does that serve your needs ? On Jul 19, 2015, at 7:30 PM, Shiyao Ma i...@introo.me wrote: Hi, I'd like to put my data selectively on some datanodes. Currently I can do that by shutting down un-needed datanodes. But this is a