On Wed, Jun 22, 2011 at 3:15 PM, Edward J. Yoon <[email protected]> wrote: > Instead removing 206 line, why don't we add dependencies as below?
I think removing line 206 is the right thing to do. Hama doesn't depend on commons-cli, hadoop-core does, and maven will properly pick that up and include the jar if allowed. The fact that Hama doesn't depend on the several items below was correctly detected by maven, in the report description of HAMA-402. You don't mention a particular problem with including the transitive dependencies, but if your problem is that it includes too many items, maven does have options to exclude a particular transitive dependency on a case-by-case basis that might be more appropriate to use. Including the dependencies below as you suggest puts Hama in the place of maintaining a transitive dependency like commons-cli in lock-step hadoop-core, where the actual dependency comes from. This is all just my $.02 worth, I'm open to arguments in favor of <excludeTransitive/>, and there is often more than one way to skin a cat with respect to dependency management. ...joe > <dependency> > <groupId>org.mortbay.jetty</groupId> > <artifactId>jetty-annotations</artifactId> > <version>6.1H.22</version> > </dependency> > <dependency> > <groupId>org.slf4j</groupId> > <artifactId>slf4j-log4j12</artifactId> > <version>1.5.2</version> > </dependency> > <dependency> > <groupId>commons-cli</groupId> > <artifactId>commons-cli</artifactId> > <version>1.2</version> > </dependency> > > Thanks. > > On Thu, Jun 23, 2011 at 7:00 AM, Edward J. Yoon <[email protected]> wrote: >> Oh... sorry, you're right. >> >> It looks like a bug. We have to fix this. >> >> [INFO] Total time: 4:36.098s >> [INFO] Finished at: Thu Jun 23 06:58:28 KST 2011 >> [INFO] Final Memory: 20M/51M >> [INFO] >> ------------------------------------------------------------------------ >> edward@test ~/workspace/hama-trunk $ ls lib/ >> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar >> jetty-util-6.1H.22.jar log4j-1.2.15.jar >> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar >> zookeeper-3.3.1.jar >> edward@tweetple ~/workspace/hama-trunk $ bin/hama zookeeper >> -bash: bin/hama: Permission denied >> edward@test ~/workspace/hama-trunk $ chmod 777 bin/** >> edward@test ~/workspace/hama-trunk $ bin/hama zookeeper >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/commons/cli/ParseException >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.commons.cli.ParseException >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:303) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) >> ... 3 more >> >> >> >> On Thu, Jun 23, 2011 at 6:45 AM, Edward J. Yoon <[email protected]> >> wrote: >>> That's so weird. >>> >>> It works well on my test cluster. >>> >>> edward@slave:~/workspace/hama-trunk$ ls lib/ >>> commons-logging-1.0.4.jar hadoop-test-0.20.2.jar >>> jetty-util-6.1H.22.jar log4j-1.2.15.jar >>> hadoop-core-0.20.2.jar jetty-6.1H.22.jar junit-3.8.1.jar >>> zookeeper-3.3.1.jar >>> edward@slave:~/workspace/hama-trunk$ bin/hama jar >>> hama-examples-0.3.0-incubating-SNAPSHOT.jar pi >>> 11/06/23 06:41:55 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir: >>> hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/submit_ebr5do >>> 11/06/23 06:41:57 INFO bsp.BSPJobClient: Running job: job_201106230640_0001 >>> 11/06/23 06:42:00 INFO bsp.BSPJobClient: Current supersteps number: 0 >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: Current supersteps number: 1 >>> 11/06/23 06:42:03 INFO bsp.BSPJobClient: The total number of supersteps: 1 >>> Estimated value of PI is 3.1534 >>> Job Finished in 7.886 seconds >>> edward@slave:~/workspace/hama-trunk$ >>> >>> On Thu, Jun 23, 2011 at 4:21 AM, Joseph Boyd >>> <[email protected]> wrote: >>>> Replying to my own post ... >>>> >>>> On Wed, Jun 22, 2011 at 11:05 AM, Joseph Boyd >>>> <[email protected]> wrote: >>>>> On Thu, Jun 16, 2011 at 4:37 AM, Ashish Agarwal <[email protected]> >>>>> wrote: >>>>>> >>>>>> When I try to run the hama daemons using the command - >>>>>> bin/hama zookeeper >>>>>> >>>>>> I get this error - >>>> >>>> It looks like the problem is actually this line in pom.xml (line 205) : >>>> <excludeTransitive>true</excludeTransitive> >>>> >>>> This prevents commons-cli (and other transitive dependencies) from >>>> being copied to the lib dir like they should be. >>>> >>>> Removing line 205 fixes the NoClassDefFoundError for me. I can open a >>>> JIRA for this after lunch, the command line is fairly broken with this >>>> line in, especially after several dependencies were (correctly) >>>> removed in HAMA-402. >>>> >>>> >>>> ...joe >>>> >>>> >>>>> I'm getting the same NoClassDefFoundError using the svn code. The >>>>> commons-cli dependency was removed recently as part of HAMA-402, and >>>>> re-adding it fixes the NoClassDefFoundError. >>>>> >>>>> I believe commons-cli should have been pulled as part of the >>>>> hadoop-core dependency, and wasn't. (though my maven skills are >>>>> rusty, and I'm no longer sure just how transitive dependencies are >>>>> pulled) >>>>> >>>>> While a quick work-around is to re-add the dependency here, the proper >>>>> fix is probably to fix the hadoop-core pom, and properly mark that it >>>>> depends on commons-cli. >>>>> >>>>> >>>>> ...joe >>>>> >>>>> >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>> org/apache/commons/cli/ParseException >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59) >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>> at org.apache.hama.ZooKeeperRunner.main(ZooKeeperRunner.java:41) >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> org.apache.commons.cli.ParseException >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >>>>>> ... 3 more >>>>>> >>>>>> >>>>>> I tried installing it a few times, but can't think of a solution to the >>>>>> problem. Can you please help ? >>>>>> >>>>>> Thank you >>>>>> >>>>>> Sincerely, >>>>>> Ashish >>>>>> >>>>>> >>>>>> On Wed, Jun 15, 2011 at 3:18 AM, Edward J. Yoon >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I would suggest following steps to debug: >>>>>>> >>>>>>> 1. Install the Hadoop 0.20.x on your laptop. >>>>>>> 2. Run the Hama deamons with 3 terminals. >>>>>>> >>>>>>> % bin/hama zookeeper >>>>>>> % bin/hama bspmaster >>>>>>> % bin/hama groom >>>>>>> >>>>>>> 3. Use the logging API to log debug. >>>>>>> For example, LOG.debug("like this"); >>>>>>> >>>>>>> On Wed, Jun 15, 2011 at 6:48 PM, Ashish Agarwal >>>>>>> <[email protected]> >>>>>>> wrote: >>>>>>> > Hi, >>>>>>> > >>>>>>> > I was wondering if anyone can give me instructions on how to debug >>>>>>> > code >>>>>>> in >>>>>>> > distributed or pseudo-distributed mode ? It would really help me. >>>>>>> > >>>>>>> > Thank you >>>>>>> > >>>>>>> > Sincerely, >>>>>>> > Ashish >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Sat, Jun 11, 2011 at 3:41 AM, Ashish Agarwal >>>>>>> > <[email protected] >>>>>>> >wrote: >>>>>>> > >>>>>>> >> >>>>>>> >> Hi, >>>>>>> >> >>>>>>> >> I have written some code for my GSoC task (Hama-367) and am now >>>>>>> >> trying >>>>>>> to >>>>>>> >> debug. >>>>>>> >> >>>>>>> >> I am trying to set up the environment into pseudo-distributed mode >>>>>>> >> so I >>>>>>> can >>>>>>> >> debug. I think the value of bsp.master.address in hama-site.xml has >>>>>>> >> to >>>>>>> be >>>>>>> >> local as this means LocalBSPRunner (as localhost would call 127.0.0.1 >>>>>>> >> server), but where do I specify that in the code in eclipse ? >>>>>>> >> >>>>>>> >> My next question is - what is the difference between MiniBSPCluster >>>>>>> >> and >>>>>>> >> LocalBSPRunner and which file should I be using to be able to step >>>>>>> through >>>>>>> >> the code ? >>>>>>> >> >>>>>>> >> These are the processes currently running on my machine - >>>>>>> >> >>>>>>> >> 6112 NameNode >>>>>>> >> 3319 ZooKeeperRunner >>>>>>> >> 3018 >>>>>>> >> 6449 SecondaryNameNode >>>>>>> >> 5709 HMaster >>>>>>> >> 5643 HQuorumPeer >>>>>>> >> 6684 TaskTracker >>>>>>> >> 6526 JobTracker >>>>>>> >> 8074 Jps >>>>>>> >> 6277 DataNode >>>>>>> >> >>>>>>> >> Thank you, >>>>>>> >> Ashish >>>>>>> >> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, Edward J. Yoon >>>>>>> @eddieyoon >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> @eddieyoon >>> >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon >
