If you want to execute a command in the background without watching logs, you can do the following:
% bin/hama zookeeper & On Thu, Jun 23, 2011 at 4:43 PM, Ashish Agarwal <[email protected]> wrote: > How long does it take to execute the command > bin/hama zookeeper > > I had to stop the execution as it stopped doing anything after some time. > > hadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ bin/hama zookeeper > 11/06/23 00:27:41 INFO server.ZooKeeperServerMain: Starting server > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14 > GMT > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:host.name=ubuntu.ubuntu-domain > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.version=1.6.0_24 > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.vendor=Sun Microsystems Inc. > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.24/jre > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.class.path=/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../target/classes:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../hama-**.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-cli-1.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/commons-logging-1.0.4.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-annotations-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/jetty-util-6.1H.22.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/junit-3.8.1.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/log4j-1.2.15.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/slf4j-log4j12-1.5.2.jar:/home/hadoop/Desktop/SecondWorkspace/hama-trunk/bin/../lib/zookeeper-3.3.1.jar > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/server:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.io.tmpdir=/tmp > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:java.compiler=<NA> > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:os.name=Linux > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server environment:os.arch=i386 > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:os.version=2.6.35-28-generic > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:user.name=hadoop > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:user.home=/home/hadoop > 11/06/23 00:27:41 INFO server.ZooKeeperServer: Server > environment:user.dir=/home/hadoop/Desktop/SecondWorkspace/hama-trunk > 11/06/23 00:27:41 INFO server.ZooKeeperServer: tickTime set to 3000 > 11/06/23 00:27:41 INFO server.ZooKeeperServer: minSessionTimeout set to -1 > 11/06/23 00:27:41 INFO server.ZooKeeperServer: maxSessionTimeout set to > 1200000 > 11/06/23 00:27:41 INFO server.NIOServerCnxn: binding to port > 0.0.0.0/0.0.0.0:21810 > 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot > /tmp/hama-hadoop/zookeeper/version-2/snapshot.0 > 11/06/23 00:27:41 INFO persistence.FileSnap: Reading snapshot > /tmp/hama-hadoop/zookeeper/version-2/snapshot.0 > 11/06/23 00:27:41 INFO persistence.FileTxnSnapLog: Snapshotting: 0 > ^Chadoop@ubuntu:~/Desktop/SecondWorkspace/hama-trunk$ > > > Thanks > Ashish > > > > On Wed, Jun 22, 2011 at 3:31 PM, Joseph Boyd > <[email protected]> wrote: >> >> On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <[email protected]> >> wrote: >> > Instead removing 206 line, why don't we add dependencies as below? >> >> I think removing line 206 is the right thing to do. Hama doesn't >> depend on commons-cli, hadoop-core does, and maven will properly pick >> that up and include the jar if allowed. >> >> The fact that Hama doesn't depend on the several items below was >> correctly detected by maven, in the report description of HAMA-402. >> >> You don't mention a particular problem with including the transitive >> dependencies, but if your problem is that it includes too many items, >> maven does have options to exclude a particular transitive dependency >> on a case-by-case basis that might be more appropriate to use. >> >> Including the dependencies below as you suggest puts Hama in the place >> of maintaining a transitive dependency like commons-cli in lock-step >> hadoop-core, where the actual dependency comes from. >> >> >> This is all just my $.02 worth, I'm open to arguments in favor of >> <excludeTransitive/>, and there is often more than one way to skin a >> cat with respect to dependency management. >> >> >> ...joe >> >> >> >> > <dependency> >> > <groupId>org.mortbay.jetty</groupId> >> > <artifactId>jetty-annotations</artifactId> >> > <version>6.1H.22</version> >> > </dependency> >> > <dependency> >> > <groupId>org.slf4j</groupId> >> > <artifactId>slf4j-log4j12</artifactId> >> > <version>1.5.2</version> >> > </dependency> >> > <dependency> >> > <groupId>commons-cli</groupId> >> > <artifactId>commons-cli</artifactId> >> > <version>1.2</version> >> > </dependency> >> > >> > Thanks. >> > >> > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <[email protected]> >> > wrote: >> >> Oh... sorry, you're right. >> >> >> >> It looks like a bug. We have to fix this. >> >> >> >> [INFO] Total time: 4:36.098s >> >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011 >> >> [INFO] Final Memory: 20M/51M >> >> [INFO] >> >> ------------------------------------------------------------------------ >> >> edward@test ~/workspace/hama-trunk $ ls lib/ >> >> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar >> >> jetty-util-6.1H.22.jar log4j-1.2.15.jar >> >> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar >> >> zookeeper-3.3.1.jar >> >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper >> >> -bash: bin/hama: Permission denied >> >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/** >> >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> >> org/apache/commons/cli/ParseException >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59) >> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> >> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41) >> >> Caused by: java.lang.ClassNotFoundException: >> >> org.apache.commons.cli.ParseException >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:303) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) >> >> ... 3 more >> >> >> >> >> >> >> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <[email protected]> >> >> wrote: >> >>> That's so weird. >> >>> >> >>> It works well on my test cluster. >> >>> >> >>> edward@slave:~/workspace/hama-trunk$ ls lib/ >> >>> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar >> >>> jetty-util-6.1H.22.jar log4j-1.2.15.jar >> >>> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar >> >>> zookeeper-3.3.1.jar >> >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar >> >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi >> >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir: >> >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do >> >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: >> >>> job_201106230640_0001 >> >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0 >> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1 >> >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: >> >>> 1 >> >>> Estimated value of PI is 3.1534 >> >>> Job Finished in 7.886 seconds >> >>> edward@slave:~/workspace/hama-trunk$ >> >>> >> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd >> >>> <[email protected]> wrote: >> >>>> Replying to my own post ... >> >>>> >> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd >> >>>> <[email protected]> wrote: >> >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal >> >>>>> <[email protected]> wrote: >> >>>>>> >> >>>>>> When I try to run the hama daemons using the command - >> >>>>>> bin/hama zookeeper >> >>>>>> >> >>>>>> I get this error - >> >>>> >> >>>> It looks like the problem is actually this line in pom.xml (line 205) : >> >>>> <excludeTransitive>true</excludeTransitive> >> >>>> >> >>>> This prevents commons-cli (and other transitive dependencies) from >> >>>> being copied to the lib dir like they should be. >> >>>> >> >>>> Removing line 205 fixes the NoClassDefFoundError for me. I can open a >> >>>> JIRA for this after lunch, the command line is fairly broken with this >> >>>> line in, especially after several dependencies were (correctly) >> >>>> removed in HAMA-402. >> >>>> >> >>>> >> >>>> ...joe >> >>>> >> >>>> >> >>>>> I'm getting the same NoClassDefFoundError using the svn code. The >> >>>>> commons-cli dependency was removed recently as part of HAMA-402, and >> >>>>> re-adding it fixes the NoClassDefFoundError. >> >>>>> >> >>>>> I believe commons-cli should have been pulled as part of the >> >>>>> hadoop-core dependency, and wasn't. (though my maven skills are >> >>>>> rusty, and I'm no longer sure just how transitive dependencies are >> >>>>> pulled) >> >>>>> >> >>>>> While a quick work-around is to re-add the dependency here, the proper >> >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it >> >>>>> depends on commons-cli. >> >>>>> >> >>>>> >> >>>>> ...joe >> >>>>> >> >>>>> >> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >> >>>>>> org/apache/commons/cli/ParseException >> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59) >> >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41) >> >>>>>> Caused by: java.lang.ClassNotFoundException: >> >>>>>> org.apache.commons.cli.ParseException >> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> >>>>>> at java.security.AccessController.doPrivileged(Native Method) >> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> >>>>>> ... 3 more >> >>>>>> >> >>>>>> >> >>>>>> I tried installing it a few times, but can't think of a solution to >> >>>>>> the >> >>>>>> problem. Can you please help ? >> >>>>>> >> >>>>>> Thank you >> >>>>>> >> >>>>>> Sincerely, >> >>>>>> Ashish >> >>>>>> >> >>>>>> >> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon >> >>>>>> <[email protected]>wrote: >> >>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> I would suggest following steps to debug: >> >>>>>>> >> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop. >> >>>>>>> 2. Run the Hama deamons with 3 terminals. >> >>>>>>> >> >>>>>>> % bin/hama zookeeper >> >>>>>>> % bin/hama bspmaster >> >>>>>>> % bin/hama groom >> >>>>>>> >> >>>>>>> 3. Use the logging API to log debug. >> >>>>>>> For example, LOG.debug("like this"); >> >>>>>>> >> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal >> >>>>>>> <[email protected]> >> >>>>>>> wrote: >> >>>>>>> > Hi, >> >>>>>>> > >> >>>>>>> > I was wondering if anyone can give me instructions on how to debug >> >>>>>>> > code >> >>>>>>> in >> >>>>>>> > distributed or pseudo-distributed mode ? It would really help me. >> >>>>>>> > >> >>>>>>> > Thank you >> >>>>>>> > >> >>>>>>> > Sincerely, >> >>>>>>> > Ashish >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal >> >>>>>>> > <[email protected] >> >>>>>>> >wrote: >> >>>>>>> > >> >>>>>>> >> >> >>>>>>> >> Hi, >> >>>>>>> >> >> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now >> >>>>>>> >> trying >> >>>>>>> to >> >>>>>>> >> debug. >> >>>>>>> >> >> >>>>>>> >> I am trying to set up the environment into pseudo-distributed >> >>>>>>> >> mode so I >> >>>>>>> can >> >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml >> >>>>>>> >> has to >> >>>>>>> be >> >>>>>>> >> local as this means LocalBSPRunner (as localhost would call >> >>>>>>> >> 127.0.0.1 >> >>>>>>> >> server), but where do I specify that in the code in eclipse ? >> >>>>>>> >> >> >>>>>>> >> My next question is - what is the difference between >> >>>>>>> >> MiniBSPCluster and >> >>>>>>> >> LocalBSPRunner and which file should I be using to be able to >> >>>>>>> >> step >> >>>>>>> through >> >>>>>>> >> the code ? >> >>>>>>> >> >> >>>>>>> >> These are the processes currently running on my machine - >> >>>>>>> >> >> >>>>>>> >> 6112 NameNode >> >>>>>>> >> 3319 ZooKeeperRunner >> >>>>>>> >> 3018 >> >>>>>>> >> 6449 SecondaryNameNode >> >>>>>>> >> 5709 HMaster >> >>>>>>> >> 5643 HQuorumPeer >> >>>>>>> >> 6684 TaskTracker >> >>>>>>> >> 6526 JobTracker >> >>>>>>> >> 8074 Jps >> >>>>>>> >> 6277 DataNode >> >>>>>>> >> >> >>>>>>> >> Thank you, >> >>>>>>> >> Ashish >> >>>>>>> >> >> >>>>>>> > >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Best Regards, Edward J. Yoon >> >>>>>>> @eddieyoon >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, Edward J. Yoon >> >>> @eddieyoon >> >>> >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> >> >> > >> > >> > >> > -- >> > Best Regards, Edward J. Yoon >> > @eddieyoon >> > > -- Best Regards, Edward J. Yoon @eddieyoon
