Now, I am having a problem starting the bspmaster :(
bin/hama bspmaster
hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama bspmaster
11/06/23 01:00:13 INFO hama.BSPMasterRunner: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting BSPMaster
STARTUP_MSG: host = ubuntu/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = -r ; compiled by 'chrisdo' on Wed Mar 10
13:22:54 PST 2010
************************************************************/
11/06/23 01:00:13 INFO bsp.BSPMaster: RPC BSPMaster: host local port 40000
11/06/23 01:00:14 FATAL hama.BSPMasterRunner:
java.net.UnknownHostException: Invalid hostname for server: local
at org.apache.hadoop.ipc.Server.bind(Server.java:198)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:253)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1026)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:488)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:450)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:441)
at org.apache.hama.bsp.BSPMaster.<init>(BSPMaster.java:248)
at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:419)
at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:413)
at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56)
11/06/23 01:00:14 INFO hama.BSPMasterRunner: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down BSPMaster at ubuntu/127.0.1.1
************************************************************/
hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
Thanks
Ashish
On Thu, Jun 23, 2011 at 12:49 AM, Edward J. Yoon <[email protected]> wrote:
> If you want to execute a command in the background without watching
> logs, you can do the following:
>
> % bin/hama zookeeper &
>
> On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <[email protected]>
> wrote:
>> How long does it take to execute the command
>> bin/hama zookeeper
>>
>> I had to stop the execution as it stopped doing anything after some time.
>>
>> hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper
>> 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14
>> GMT
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:host.name=ubuntu.ubuntu-domain
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.version=1.6.0_24
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.vendor=Sun Microsystems Inc.
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.io.tmpdir=/tmp
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:java.compiler=<NA>
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:os.name=Linux
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:os.arch=i386
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:os.version=2.6.35-28-generic
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.name=hadoop
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.home=/home/hadoop
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server
>> environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1
>> 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to
>> 1200000
>> 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port
>> 0.0.0.0/0.0.0.0:21810
>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot
>> /tmp/hama-hadoop/zookeeper/version-2/snapshot.0
>> 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0
>> ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$
>>
>>
>> Thanks
>> Ashish
>>
>>
>>
>> On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd
>> <[email protected]> wrote:
>>>
>>> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <[email protected]>
>>> wrote:
>>> > Instead removing 206 line, why don't we add dependencies as below?
>>>
>>> I think removing line 206 is the right thing to do. Hama doesn't
>>> depend on commons-cli, hadoop-core does, and maven will properly pick
>>> that up and include the jar if allowed.
>>>
>>> The fact that Hama doesn't depend on the several items below was
>>> correctly detected by maven, in the report description of HAMA-402.
>>>
>>> You don't mention a particular problem with including the transitive
>>> dependencies, but if your problem is that it includes too many items,
>>> maven does have options to exclude a particular transitive dependency
>>> on a case-by-case basis that might be more appropriate to use.
>>>
>>> Including the dependencies below as you suggest puts Hama in the place
>>> of maintaining a transitive dependency like commons-cli in lock-step
>>> hadoop-core, where the actual dependency comes from.
>>>
>>>
>>> This is all just my $.02 worth, I'm open to arguments in favor of
>>> <excludeTransitive/>, and there is often more than one way to skin a
>>> cat with respect to dependency management.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>> > <dependency>
>>> > <groupId>org.mortbay.jetty</groupId>
>>> > <artifactId>jetty-annotations</artifactId>
>>> > <version>6.1H.22</version>
>>> > </dependency>
>>> > <dependency>
>>> > <groupId>org.slf4j</groupId>
>>> > <artifactId>slf4j-log4j12</artifactId>
>>> > <version>1.5.2</version>
>>> > </dependency>
>>> > <dependency>
>>> > <groupId>commons-cli</groupId>
>>> > <artifactId>commons-cli</artifactId>
>>> > <version>1.2</version>
>>> > </dependency>
>>> >
>>> > Thanks.
>>> >
>>> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <[email protected]>
>>> > wrote:
>>> >> Oh... sorry, you're right.
>>> >>
>>> >> It looks like a bug. We have to fix this.
>>> >>
>>> >> [INFO] Total time: 4:36.098s
>>> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011
>>> >> [INFO] Final Memory: 20M/51M
>>> >> [INFO]
>>> >> ------------------------------------------------------------------------
>>> >> edward@test ~/workspace/hama-trunk $ ls lib/
>>> >> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar
>>> >> jetty-util-6.1H.22.jar log4j-1.2.15.jar
>>> >> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar
>>> >> zookeeper-3.3.1.jar
>>> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >> -bash: bin/hama: Permission denied
>>> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/**
>>> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper
>>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >> org/apache/commons/cli/ParseException
>>> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >> Caused by: java.lang.ClassNotFoundException:
>>> >> org.apache.commons.cli.ParseException
>>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>> >> at java.security.AccessController.doPrivileged(Native Method)
>>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
>>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
>>> >> ... 3 more
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <[email protected]>
>>> >> wrote:
>>> >>> That's so weird.
>>> >>>
>>> >>> It works well on my test cluster.
>>> >>>
>>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>>> >>> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar
>>> >>> jetty-util-6.1H.22.jar log4j-1.2.15.jar
>>> >>> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar
>>> >>> zookeeper-3.3.1.jar
>>> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar
>>> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi
>>> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
>>> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do
>>> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job:
>>> >>> job_201106230640_0001
>>> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of
>>> >>> supersteps: 1
>>> >>> Estimated value of PI is 3.1534
>>> >>> Job Finished in 7.886 seconds
>>> >>> edward@slave:~/workspace/hama-trunk$
>>> >>>
>>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd
>>> >>> <[email protected]> wrote:
>>> >>>> Replying to my own post ...
>>> >>>>
>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd
>>> >>>> <[email protected]> wrote:
>>> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal
>>> >>>>> <[email protected]> wrote:
>>> >>>>>>
>>> >>>>>> When I try to run the hama daemons using the command -
>>> >>>>>> bin/hama zookeeper
>>> >>>>>>
>>> >>>>>> I get this error -
>>> >>>>
>>> >>>> It looks like the problem is actually this line in pom.xml (line 205)
>>> >>>> :
>>> >>>> <excludeTransitive>true</excludeTransitive>
>>> >>>>
>>> >>>> This prevents commons-cli (and other transitive dependencies) from
>>> >>>> being copied to the lib dir like they should be.
>>> >>>>
>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me. I can open a
>>> >>>> JIRA for this after lunch, the command line is fairly broken with this
>>> >>>> line in, especially after several dependencies were (correctly)
>>> >>>> removed in HAMA-402.
>>> >>>>
>>> >>>>
>>> >>>> ...joe
>>> >>>>
>>> >>>>
>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code. The
>>> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and
>>> >>>>> re-adding it fixes the NoClassDefFoundError.
>>> >>>>>
>>> >>>>> I believe commons-cli should have been pulled as part of the
>>> >>>>> hadoop-core dependency, and wasn't. (though my maven skills are
>>> >>>>> rusty, and I'm no longer sure just how transitive dependencies are
>>> >>>>> pulled)
>>> >>>>>
>>> >>>>> While a quick work-around is to re-add the dependency here, the proper
>>> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it
>>> >>>>> depends on commons-cli.
>>> >>>>>
>>> >>>>>
>>> >>>>> ...joe
>>> >>>>>
>>> >>>>>
>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> >>>>>> org/apache/commons/cli/ParseException
>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
>>> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41)
>>> >>>>>> Caused by: java.lang.ClassNotFoundException:
>>> >>>>>> org.apache.commons.cli.ParseException
>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >>>>>> ... 3 more
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I tried installing it a few times, but can't think of a solution to
>>> >>>>>> the
>>> >>>>>> problem. Can you please help ?
>>> >>>>>>
>>> >>>>>> Thank you
>>> >>>>>>
>>> >>>>>> Sincerely,
>>> >>>>>> Ashish
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon
>>> >>>>>> <[email protected]>wrote:
>>> >>>>>>
>>> >>>>>>> Hi,
>>> >>>>>>>
>>> >>>>>>> I would suggest following steps to debug:
>>> >>>>>>>
>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop.
>>> >>>>>>> 2. Run the Hama deamons with 3 terminals.
>>> >>>>>>>
>>> >>>>>>> % bin/hama zookeeper
>>> >>>>>>> % bin/hama bspmaster
>>> >>>>>>> % bin/hama groom
>>> >>>>>>>
>>> >>>>>>> 3. Use the logging API to log debug.
>>> >>>>>>> For example, LOG.debug("like this");
>>> >>>>>>>
>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal
>>> >>>>>>> <[email protected]>
>>> >>>>>>> wrote:
>>> >>>>>>> > Hi,
>>> >>>>>>> >
>>> >>>>>>> > I was wondering if anyone can give me instructions on how to
>>> >>>>>>> > debug code
>>> >>>>>>> in
>>> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me.
>>> >>>>>>> >
>>> >>>>>>> > Thank you
>>> >>>>>>> >
>>> >>>>>>> > Sincerely,
>>> >>>>>>> > Ashish
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal
>>> >>>>>>> > <[email protected]
>>> >>>>>>> >wrote:
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >> Hi,
>>> >>>>>>> >>
>>> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now
>>> >>>>>>> >> trying
>>> >>>>>>> to
>>> >>>>>>> >> debug.
>>> >>>>>>> >>
>>> >>>>>>> >> I am trying to set up the environment into pseudo-distributed
>>> >>>>>>> >> mode so I
>>> >>>>>>> can
>>> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml
>>> >>>>>>> >> has to
>>> >>>>>>> be
>>> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call
>>> >>>>>>> >> 127.0.0.1
>>> >>>>>>> >> server), but where do I specify that in the code in eclipse ?
>>> >>>>>>> >>
>>> >>>>>>> >> My next question is - what is the difference between
>>> >>>>>>> >> MiniBSPCluster and
>>> >>>>>>> >> LocalBSPRunner and which file should I be using to be able to
>>> >>>>>>> >> step
>>> >>>>>>> through
>>> >>>>>>> >> the code ?
>>> >>>>>>> >>
>>> >>>>>>> >> These are the processes currently running on my machine -
>>> >>>>>>> >>
>>> >>>>>>> >> 6112 NameNode
>>> >>>>>>> >> 3319 ZooKeeperRunner
>>> >>>>>>> >> 3018
>>> >>>>>>> >> 6449 SecondaryNameNode
>>> >>>>>>> >> 5709 HMaster
>>> >>>>>>> >> 5643 HQuorumPeer
>>> >>>>>>> >> 6684 TaskTracker
>>> >>>>>>> >> 6526 JobTracker
>>> >>>>>>> >> 8074 Jps
>>> >>>>>>> >> 6277 DataNode
>>> >>>>>>> >>
>>> >>>>>>> >> Thank you,
>>> >>>>>>> >> Ashish
>>> >>>>>>> >>
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Best Regards, Edward J. Yoon
>>> >>>>>>> @eddieyoon
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Best Regards, Edward J. Yoon
>>> >>> @eddieyoon
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Edward J. Yoon
>>> >> @eddieyoon
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>> >
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
