This occurred again.! While running the big setup of BAM, one hadoop datanode somehow failed. It seems the job submitted to that node got halted for several hours, due to that one BAM analyzer node too logged that one map-reduce job was running forever (not reducing).
After that when I try to restart again the hadoop cluster the above discussed problem occurred.! On Thu, Aug 9, 2012 at 8:52 AM, Deependra Ariyadewa <[email protected]> wrote: > > > On Wed, Aug 8, 2012 at 3:26 PM, Chamara Ariyarathne <[email protected]>wrote: > >> >> >> On Wed, Aug 8, 2012 at 3:15 PM, Charitha Kankanamge <[email protected]>wrote: >> >>> These configuration settings, deployment tips/tricks everything has to >>> be documented somewhere. Specially, in a product like BAM, since there are >>> so many configuration options, we should provide users with a deployment >>> script to setup a BAM cluster. >> >> +1 >> https://wso2.org/jira/browse/BAM-695 >> > > Creating one script will not be an easy task but creating set of > preconfigured VM would be possible. > > Thanks, > > Deependra. > >> >>> >>> /Charitha >>> >>> On Wed, Aug 8, 2012 at 10:57 AM, Chamara Ariyarathne >>> <[email protected]>wrote: >>> >>>> This got fixed when the directory referring with hadoop.tmp.dir in >>>> core-site.xml is cleaned up and then the namenode is formatted. >>>> >>>> >>>> On Wed, Aug 8, 2012 at 10:25 AM, Deependra Ariyadewa <[email protected]>wrote: >>>> >>>>> Hi Chamara, >>>>> >>>>> Pl try to start the name node without all the other services and >>>>> attach the name node log. It will help to isolate the issue in Name Node. >>>>> >>>>> You can start the name node by executing "bin/hadoop namenode" >>>>> >>>>> >>>>> Thanks, >>>>> Deependra. >>>>> >>>>> On Wed, Aug 8, 2012 at 9:56 AM, Chamara Ariyarathne <[email protected] >>>>> > wrote: >>>>> >>>>>> I get the following [1] when starting hadoop namenode in the >>>>>> distributed hadoop cluster and it prevents from starting all the hadoop >>>>>> nodes. >>>>>> >>>>>> /etc/hosts is: >>>>>> 10.100.3.221 bamhadoop01 >>>>>> 10.100.3.228 bamhadoop02 >>>>>> 10.100.3.224 bamhadoop03 >>>>>> 10.100.3.244 appserver.cloud-test.wso2.com >>>>>> >>>>>> # The following lines are desirable for IPv6 capable hosts >>>>>> ::1 ip6-localhost ip6-loopback >>>>>> fe00::0 ip6-localnet >>>>>> ff00::0 ip6-mcastprefix >>>>>> ff02::1 ip6-allnodes >>>>>> ff02::2 ip6-allrouters >>>>>> >>>>>> $ hostname >>>>>> bamhadoop01 >>>>>> >>>>>> Property in core-site.xml >>>>>> <property> >>>>>> <name>fs.default.name</name> >>>>>> <value>hdfs://bamhadoop01:9000</value> >>>>>> </property> >>>>>> <property> >>>>>> >>>>>> Can ssh to any other hadoop datanode machines without password. >>>>>> >>>>>> [1] >>>>>> 2012-08-08 09:45:05,289 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >>>>>> hadoop-metrics2.properties >>>>>> 2012-08-08 09:45:05,297 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> MetricsSystem,sub=Stats registered. >>>>>> 2012-08-08 09:45:05,297 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >>>>>> period at 10 second(s). >>>>>> 2012-08-08 09:45:05,297 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics >>>>>> system started >>>>>> 2012-08-08 09:45:05,478 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> ugi >>>>>> registered. >>>>>> 2012-08-08 09:45:05,481 WARN >>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi >>>>>> already >>>>>> exists! >>>>>> 2012-08-08 09:45:05,482 INFO >>>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>>>>> Updating the current master key for generating delegation tokens >>>>>> 2012-08-08 09:45:05,483 INFO >>>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>>>>> Starting expired delegation token remover thread, >>>>>> tokenRemoverScanInterval=60 min(s) >>>>>> 2012-08-08 09:45:05,483 INFO >>>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >>>>>> Updating the current master key for generating delegation tokens >>>>>> 2012-08-08 09:45:05,483 INFO org.apache.hadoop.mapred.JobTracker: >>>>>> Scheduler configured with (memSizeForMapSlotOnJT, >>>>>> memSizeForReduceSlotOnJT, >>>>>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) >>>>>> 2012-08-08 09:45:05,484 INFO org.apache.hadoop.util.HostsFileReader: >>>>>> Refreshing hosts (include/exclude) list >>>>>> 2012-08-08 09:45:05,517 INFO org.apache.hadoop.mapred.JobTracker: >>>>>> Starting jobtracker with owner as bamtesttmp >>>>>> 2012-08-08 09:45:05,531 INFO org.apache.hadoop.ipc.Server: Starting >>>>>> SocketReader >>>>>> 2012-08-08 09:45:05,533 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> RpcDetailedActivityForPort9001 registered. >>>>>> 2012-08-08 09:45:05,534 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> RpcActivityForPort9001 registered. >>>>>> 2012-08-08 09:45:10,598 INFO org.mortbay.log: Logging to >>>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >>>>>> org.mortbay.log.Slf4jLog >>>>>> 2012-08-08 09:45:10,634 INFO org.apache.hadoop.http.HttpServer: Added >>>>>> global filtersafety >>>>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >>>>>> 2012-08-08 09:45:10,655 INFO org.apache.hadoop.http.HttpServer: Port >>>>>> returned by webServer.getConnectors()[0].getLocalPort() before open() is >>>>>> -1. Opening the listener on 50030 >>>>>> 2012-08-08 09:45:10,656 INFO org.apache.hadoop.http.HttpServer: >>>>>> listener.getLocalPort() returned 50030 >>>>>> webServer.getConnectors()[0].getLocalPort() returned 50030 >>>>>> 2012-08-08 09:45:10,656 INFO org.apache.hadoop.http.HttpServer: Jetty >>>>>> bound to port 50030 >>>>>> 2012-08-08 09:45:10,656 INFO org.mortbay.log: jetty-6.1.26 >>>>>> 2012-08-08 09:45:10,676 WARN org.mortbay.log: Can't reuse >>>>>> /tmp/Jetty_0_0_0_0_50030_job____yn7qmk, using >>>>>> /tmp/Jetty_0_0_0_0_50030_job____yn7qmk_2518359383055609936 >>>>>> 2012-08-08 09:45:10,811 INFO org.mortbay.log: Started >>>>>> [email protected]:50030 >>>>>> 2012-08-08 09:45:10,815 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> jvm >>>>>> registered. >>>>>> 2012-08-08 09:45:10,815 INFO >>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >>>>>> JobTrackerMetrics registered. >>>>>> 2012-08-08 09:45:10,815 INFO org.apache.hadoop.mapred.JobTracker: >>>>>> JobTracker up at: 9001 >>>>>> 2012-08-08 09:45:10,815 INFO org.apache.hadoop.mapred.JobTracker: >>>>>> JobTracker webserver: 50030 >>>>>> 2012-08-08 09:45:11,867 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 0 >>>>>> time(s). >>>>>> 2012-08-08 09:45:12,868 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 1 >>>>>> time(s). >>>>>> 2012-08-08 09:45:13,868 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 2 >>>>>> time(s). >>>>>> 2012-08-08 09:45:14,869 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 3 >>>>>> time(s). >>>>>> 2012-08-08 09:45:15,869 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 4 >>>>>> time(s). >>>>>> 2012-08-08 09:45:16,870 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 5 >>>>>> time(s). >>>>>> 2012-08-08 09:45:17,870 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 6 >>>>>> time(s). >>>>>> 2012-08-08 09:45:18,871 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 7 >>>>>> time(s). >>>>>> 2012-08-08 09:45:19,871 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 8 >>>>>> time(s). >>>>>> 2012-08-08 09:45:20,872 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 9 >>>>>> time(s). >>>>>> 2012-08-08 09:45:20,874 INFO org.apache.hadoop.mapred.JobTracker: >>>>>> problem cleaning system directory: null >>>>>> java.net.ConnectException: Call to bamhadoop01/10.100.3.221:9000failed >>>>>> on connection exception: java.net.ConnectException: Connection >>>>>> refused >>>>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1057) >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1033) >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) >>>>>> at $Proxy5.getProtocolVersion(Unknown Source) >>>>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364) >>>>>> at >>>>>> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) >>>>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208) >>>>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175) >>>>>> at >>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) >>>>>> at >>>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1310) >>>>>> at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65) >>>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328) >>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) >>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109) >>>>>> at org.apache.hadoop.mapred.JobTracker$3.run(JobTracker.java:2349) >>>>>> at org.apache.hadoop.mapred.JobTracker$3.run(JobTracker.java:2347) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at javax.security.auth.Subject.doAs(Subject.java:396) >>>>>> at >>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) >>>>>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2347) >>>>>> at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2171) >>>>>> at >>>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300) >>>>>> at >>>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291) >>>>>> at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4956) >>>>>> Caused by: java.net.ConnectException: Connection refused >>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>>>> at >>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) >>>>>> at >>>>>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) >>>>>> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:406) >>>>>> at >>>>>> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:414) >>>>>> at >>>>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:527) >>>>>> at >>>>>> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:187) >>>>>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164) >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1010) >>>>>> ... 22 more >>>>>> 2012-08-08 09:45:31,877 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 0 >>>>>> time(s). >>>>>> 2012-08-08 09:45:32,878 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 1 >>>>>> time(s). >>>>>> 2012-08-08 09:45:33,878 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 2 >>>>>> time(s). >>>>>> 2012-08-08 09:45:34,879 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 3 >>>>>> time(s). >>>>>> 2012-08-08 09:45:35,879 INFO org.apache.hadoop.ipc.Client: Retrying >>>>>> connect to server: bamhadoop01/10.100.3.221:9000. Already tried 4 >>>>>> time(s). >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chamara Ariyarathne* >>>>>> Senior Software Engineer - QA; >>>>>> WSO2 Inc; http://www.wso2.com/. >>>>>> Mobile; *+94772786766* >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Deependra Ariyadewa >>>>> WSO2, Inc. http://wso2.com/ http://wso2.org >>>>> >>>>> email [email protected]; cell +94 71 403 5996 ; >>>>> Blog http://risenfall.wordpress.com/ >>>>> PGP info: KeyID: 'DC627E6F' >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Chamara Ariyarathne* >>>> Senior Software Engineer - QA; >>>> WSO2 Inc; http://www.wso2.com/. >>>> Mobile; *+94772786766* >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >> >> >> -- >> *Chamara Ariyarathne* >> Senior Software Engineer - QA; >> WSO2 Inc; http://www.wso2.com/. >> Mobile; *+94772786766* >> >> > > > -- > Deependra Ariyadewa > WSO2, Inc. http://wso2.com/ http://wso2.org > > email [email protected]; cell +94 71 403 5996 ; > Blog http://risenfall.wordpress.com/ > PGP info: KeyID: 'DC627E6F' > > -- *Chamara Ariyarathne* Senior Software Engineer - QA; WSO2 Inc; http://www.wso2.com/. Mobile; *+94772786766*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
