Using 2.4 hadoop version - Thinking in a future deployment in AWS (EC2)

2015-03-28 Thread José Luis Larroque
Hi people, i'm new on this user list, and i need help! I'm starting with hadoop recently, and i'm trying to use it with giraph. So, for building giraph, i choose the 2.4 release of hadoop (because according to this link

Re: How to know when datanode are marked dead by namenode

2015-03-28 Thread Himawan Mahardianto
So, if I want to change the default dead time, I have to compile from source first? Is there any other way that I use to change dead time from native hadoop-2.6 (not from source)? Or maybe the native doesn't have a feature to change the dead time? Thank's all for the responses :) On Sun, Mar 29,

Does Hadoop 2.6.0 have job level blacklisting?

2015-03-28 Thread Chris Mawata
At http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers is a description of how you can have a script check the health of a node and indicate to the ResourceManager that it is unhealthy. This seems to be at the cluster level.

Re: How to know when datanode are marked dead by namenode

2015-03-28 Thread Himawan Mahardianto
Thank you for your response and explanation, but I couldn't find dfs.namenode.heartbeat.recheck-interval parameter on http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml that you told before, maybe any other formula with this version? On Sat, Mar 28, 2015 at

Re: How to know when datanode are marked dead by namenode

2015-03-28 Thread Ted Yu
Himawan: Please see the following constants in hadoop-hdfs-project//hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java : public static final String DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY = dfs.namenode.heartbeat.recheck-interval; public static final int

Re: How to know when datanode are marked dead by namenode

2015-03-28 Thread Ted Yu
Himawan: You don't need to recompile the code. Please see this thread http://search-hadoop.com/m/LgpTk2FkzEk The last comment in that thread led to: https://issues.apache.org/jira/browse/HDFS-7685 Cheers On Sat, Mar 28, 2015 at 6:36 PM, Himawan Mahardianto mahardia...@ugm.ac.id wrote: So, if

RE: How to know when datanode are marked dead by namenode

2015-03-28 Thread Brahma Reddy Battula
HI The NameNode updates this detail after 10.30 minutes by default. You can see the dead and live datanodes at that time. It computes this heartbeatExpireInterval time by the following formula heartbeatExpireInterval = 2 * heartbeatRecheckInterval + 10 * 1000* heartbeatInterval where