Re: While starting HDFS process getting stucked.

Stephan Gammeter Fri, 16 Sep 2011 02:10:12 -0700

Are your HDFS nodes running? Did they complete the startup? What do the logs 
say?

On machines where /dev/random ist starved (machines with not much load and maybe VMs) i think there can be an issue with jetty (internal http server) thatblocks during startup, because it wants to initialize the secure random number generator.


if you see in your datanode logs, that they get stuck upon startup:

stephaga@googolplex:/home/awesome/hadoop/hadoop $ head -n 30 
logs/hadoop-awesome-datanode-bender15.log.2011-09-07
2011-09-07 16:47:11,712 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = bender15##################
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2-append
STARTUP_MSG:   build = 
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append -r 
1057313; compiled by 'awesome' on Fri Feb 18 15:36:52 CET 2011
************************************************************/
2011-09-07 16:47:19,051 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Registered FSDatasetStatusMBean
2011-09-07 16:47:19,054 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Opened info server at 50010
2011-09-07 16:47:19,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
Balancing bandwith is 16777216 bytes/s
2011-09-07 16:47:19,118 INFO org.mortbay.log: Logging to 
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening thelistener on 50075

2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: 
listener.getLocalPort() returned 50075 
webServer.getConnectors()[0].getLocalPort() returned 50075
2011-09-07 16:47:19,191 INFO org.apache.hadoop.http.HttpServer: Jetty bound to 
port 50075
2011-09-07 16:47:19,191 INFO org.mortbay.log: jetty-6.1.14
----> STUCK HERE

then try adding the following line to your "hadoop-env.sh" :

# cf: http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup
# cf: 
http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom
export HADOOP_OPTS="-Djava.security.egd=file:/dev/./urandom"
#

hope it helps,

best,
Stephan

On 09/16/2011 10:54 AM, kiranprasad wrote:

Yes I ve formatted the namenode.
*From:* Sudharsan Sampath <mailto:sudha...@gmail.com>
*Sent:* Friday, September 16, 2011 2:11 PM
*To:* hdfs-user@hadoop.apache.org <mailto:hdfs-user@hadoop.apache.org>
*Subject:* Re: While starting HDFS process getting stucked.
Have u formatted ur namenode ?
Thanks
Sudhan S

On Fri, Sep 16, 2011 at 11:01 AM, kiranprasad <kiranprasa...@imimobile.com 
<mailto:kiranprasa...@imimobile.com>> wrote:

    Hi

    I am new to Hadoop and PIG,

    For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves)

    I ve installed PIG in 10.0.0.61 VM.=20

    Hadoop version : hadoop-0.20.2 and PIG : pig-0.8.1
    I ve updated the xmls , please find the below

    mapred site.xml
    --------------
    <configuration>
    <property>
    <name>mapred.job.tracker</**name>
    <value>10.0.0.61:8021 <http://10.0.0.61:8021></value>
    </property>
    </configuration>


    core-site.xml
    ----------
    <configuration>
    <property>
    <name>fs.default.name <http://fs.default.name></name>
    <value>hdfs://10.0.0.61:8020 <http://10.0.0.61:8020></**value>

    </property>
    </configuration>

    Hdfs-site.xml
    ----------------
    <configuration>
    <property>
    <name>fs.default.name <http://fs.default.name></name>
    <value>hdfs://10.0.0.61:8020 <http://10.0.0.61:8020></**value>
    </property>
    <property>
    <name>mapred.job.tracker</**name>
    <value>10.0.0.61:8021 <http://10.0.0.61:8021></value>

    </property>
    <property>
    <name>dfs.replication</name>
    <value>2</value>
    </property>

    masters
    ---------
    10.0.0.61

    slaves
    --------

    10.0.0.62
    10.0.0.63


    I ve tried with hadoop fs -ls but still facing the same problem.

    [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/start-all.sh
    starting namenode, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-nameno=
    de-pig4.out
    10.0.0.62 <http://10.0.0.62>: starting datanode, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-datano=
    de-pig3.out
    10.0.0.63 <http://10.0.0.63>: starting datanode, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-datano=
    de-pig2.out
    10.0.0.61 <http://10.0.0.61>: starting secondarynamenode, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-second=
    arynamenode-pig4.out
    starting jobtracker, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-jobtra=
    cker-pig4.out
    10.0.0.63 <http://10.0.0.63>: starting tasktracker, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-tasktr=
    acker-pig2.out
    10.0.0.62 <http://10.0.0.62>: starting tasktracker, logging to
    /home/kiranprasad.g/hadoop-0.20.2/bin/../logs/hadoop-kiranprasad.g-tasktr=
    acker-pig3.out
    [kiranprasad.g@pig4 hadoop-0.20.2]$
    [kiranprasad.g@pig4 hadoop-0.20.2]$
    [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls

    After this it stopped running, it got stucked.

    Regards
    Kiran.G

Re: While starting HDFS process getting stucked.

Reply via email to