How does AWS know how many map/reduce slot should be configured to each EC2 instance?

Hi All We have a plan to move to Amazon AWS cloud, by doing some research i find that i can start the map/reduce cluster in AWS with the following command:% bin/hadoop-ec2 launch-cluster test-cluster 2 The command allows me to start a cluster with required nodes(no more than 20, correct me if i

Re: How does AWS know how many map/reduce slot should be configured to each EC2 instance?

1. Yes, it's depends on instance type. Generally, number of map slots + number of reduce slots = number of ECU, number of map slots / number of reduce slots = 3. You can customize these numbers. 2. Yes, Number of nodes * Running hours * Price per EMR node per hour (EMR node is a little bit more

RE: How does AWS know how many map/reduce slot should be configured to each EC2 instance?

Hi Tianyi Thanks for the reply, that's really help. So i have two further questions: 1. You said i can customize the number of the slots on AWS, how to do it? i know i can do it in the mapred-site.xml if i created the cluster without AWS.2. You mentioned about the EMR node, will the hadoop-ec2

Re:

Hello Tariq, I solved the problem. There must have been some problem in the custom input format i created. so i took a sample custom input format which was working in cdh4 environment and applied the changes as per my requirement. It is working now. But i havent tested that code in apache hadoop

Re: How does AWS know how many map/reduce slot should be configured to each EC2 instance?

Hey, On 19 Jul 2013, at 07:55, WangRamon ramon_w...@hotmail.com wrote: Hi Tianyi Thanks for the reply, that's really help. So i have two further questions: 1. You said i can customize the number of the slots on AWS, how to do it? i know i can do it in the mapred-site.xml if i created

Re:

Glad to hear that :) Warm Regards, Tariq cloudfront.blogspot.com On Fri, Jul 19, 2013 at 1:10 PM, Anit Alexander anitama...@gmail.comwrote: Hello Tariq, I solved the problem. There must have been some problem in the custom input format i created. so i took a sample custom input format which

./hdfs namenode -bootstrapStandby error

I use hadoop-2.0.5 version and use QJM for HA. I use ./hdfs namenode -bootstrapStandby for StandbyNameNode, but report below error: = About to bootstrap Standby ID nn2 from: Nameservice ID: mycluster Other Namenode ID: nn1

DistributedCache incompatibility issue between 1.0 and 2.0

Hi, I wonder why setLocalFiles and addLocalFiles methods have been removed, and what should I use instead of them? -- Best Regards, Edward J. Yoon @eddieyoon

Unexpected problem in creating temporary file

Hi, I am seeing many such errors on a datanode - 2013-07-18 22:10:49,473 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.254.0.40:50010, storageID=DS-595314104-10.254.0.40-50010-1374154266946, infoPort=50075, ipcPort=50020):DataXceiver java.io.IOException:

RE: DistributedCache incompatibility issue between 1.0 and 2.0

I have been using Job.addCacheFile() to cache files in the distributed cache. It has been working for me on Hadoop 2.0.5: public void addCacheFile(URI uri) Add a file to be localized Parameters: uri - The uri of the cache to be localized -Original Message- From: Edward J. Yoon

Re: Namenode automatically going to safemode with 2.1.0-beta

Hi Harsh, I have made my dfs.namenode.name.dir point to a subdirectory of my home, and I don't see this issue again. So, is this a bug that we need to log into JIRA? Thanks, Kishore On Tue, Jul 16, 2013 at 6:39 AM, Harsh J ha...@cloudera.com wrote: 2013-07-12 11:04:26,002 WARN

Re: DistributedCache incompatibility issue between 1.0 and 2.0

See this thread also: http://search-hadoop.com/m/3pgakkVpm71/Distributed+Cache+omkarsubj=Re+Distributed+Cache On Fri, Jul 19, 2013 at 6:20 AM, Botelho, Andrew andrew.bote...@emc.comwrote: I have been using Job.addCacheFile() to cache files in the distributed cache. It has been working for me

Re: ./hdfs namenode -bootstrapStandby error

hi, can you using 'hdfs namenode -initializeSharedEdits' on the active NN, remember start all journal nodes before try this. On Jul 19, 2013 5:17 PM, lei liu liulei...@gmail.com wrote: I use hadoop-2.0.5 version and use QJM for HA. I use ./hdfs namenode -bootstrapStandby for

Re: Namenode automatically going to safemode with 2.1.0-beta

this is not a bug. it has been documented. On Jul 19, 2013 10:13 PM, Krishna Kishore Bonagiri write2kish...@gmail.com wrote: Hi Harsh, I have made my dfs.namenode.name.dir point to a subdirectory of my home, and I don't see this issue again. So, is this a bug that we need to log into

Re: Namenode automatically going to safemode with 2.1.0-beta

Yeah I believe your /tmp was probably misbehaving somehow (running out of space or otherwise). You could log a JIRA for the null seen in the log though, it shouldn't have done that and should've shown the real mount point. On Fri, Jul 19, 2013 at 8:47 PM, Azuryy Yu azury...@gmail.com wrote: this

Re: DistributedCache incompatibility issue between 1.0 and 2.0

check https://issues.apache.org/jira/browse/MAPREDUCE-4493 and https://issues.apache.org/jira/browse/YARN-916 Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Fri, Jul 19, 2013 at 8:12 AM, Ted Yu yuzhih...@gmail.com wrote: See this thread also:

Re: Unexpected problem in creating temporary file

Any suggestion ? I am stuck. Regards, Ajay Srivastava On 19-Jul-2013, at 5:54 PM, Ajay Srivastava wrote: Hi, I am seeing many such errors on a datanode - 2013-07-18 22:10:49,473 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.254.0.40:50010,

Wish to subscribe

Regards Pradeep Singh

Wish to subscribe

Regards Pradeep Singh

subsicrbe

Regards Pradeep Singh

RE: subsicrbe

Hi Pradeep, Please send mail to subscribe mail ids, after subscription if you have any queries you can reach to the corresponding lists. You can find the subscribe mail ids in this page. http://hadoop.apache.org/mailing_lists.html Thanks Devaraj k From: Pradeep Singh