yeah. i tracked the problem to what i mentioned earlier. ZK is run with the wrong jar when using -libjars. I have to figure out what's the expected behavior though, because the logic is kind obscure in the code.
On Mon, Aug 26, 2013 at 11:24 PM, Kyle Orlando <[email protected]>wrote: > Yeah, exactly. We couldn't get it to work otherwise. > > > On Mon, Aug 26, 2013 at 11:00 AM, Claudio Martella < > [email protected]> wrote: > >> you mean by running zookeeper independently? >> >> >> On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando >> <[email protected]>wrote: >> >>> We were also experiencing similar problems when specifying -libjars as >>> opposed to just using a fat jar. I believe we fixed it by setting the >>> giraph.zkList property, but this only appears to work when we list one node >>> as a zookeeper. >>> >>> >>> On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella < >>> [email protected]> wrote: >>> >>>> OK, then I'm going to open an issue for that. >>>> >>>> >>>> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium < >>>> [email protected]> wrote: >>>> >>>>> Yes for the zookeeper problem I passed a seperate jar through -libjars >>>>> command. If I use additional jars zookeeper fails. >>>>> On Aug 26, 2013 2:51 PM, "Claudio Martella" < >>>>> [email protected]> wrote: >>>>> >>>>>> there must be a misunderstanding. i was referring to the zookeeper >>>>>> problem. >>>>>> >>>>>> >>>>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> No. I added my files(it was just a copy of one of the example >>>>>>> program to a different package) to the jar files of giraph. But it was >>>>>>> still giving me classNotFoundException. Can you give me some simple >>>>>>> example >>>>>>> program with instructions on how to deploy it. So I can start playing >>>>>>> with >>>>>>> giraph and make changes to the program and learn, then start working on >>>>>>> my >>>>>>> project in giraph. >>>>>>> I will be very thankful if you can help me with this. >>>>>>> >>>>>>> Thanking you >>>>>>> -Vivek Sembium >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> but you were still using an additional jar added through -libjars, >>>>>>>> right? >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> @Claudio Martella Your solution didnt work either. I basically >>>>>>>>> tried copying the pageRankBenchmark to my own package, renamed the >>>>>>>>> package. >>>>>>>>> It compiles fine with giraph. But I couldnt run it even if I add those >>>>>>>>> files to giraph jar before deployment. Help? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> you have this problem when you use two jars (one with giraph and >>>>>>>>>> one with your classes) instead of a single fat-jar, correct? I >>>>>>>>>> tracked the >>>>>>>>>> same problem a few weeks ago, basically zookeeper is run passing the >>>>>>>>>> wrong >>>>>>>>>> jar. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you for your suggestion. It worked. Its not giving class >>>>>>>>>>> not found exception. But its giving me a new error >>>>>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found >>>>>>>>>>> that its unable to connect to zookeeper service. >>>>>>>>>>> >>>>>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable >>>>>>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>>>>>>> at >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>>>>>>>>> Caused by: java.lang.IllegalStateException: >>>>>>>>>>> onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>>>> at >>>>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727) >>>>>>>>>>> at >>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371) >>>>>>>>>>> at >>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204) >>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59) >>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89) >>>>>>>>>>> ... 7 more >>>>>>>>>>> >>>>>>>>>>> Immediately I ran page rank benchmark and it executed >>>>>>>>>>> successfully both from giraph in lib directory and also from >>>>>>>>>>> giraphs own >>>>>>>>>>> directory. >>>>>>>>>>> >>>>>>>>>>> Can you give me a very simple java program(finding maximum in a >>>>>>>>>>> graph or simple page rank program) in giraph along with its jar >>>>>>>>>>> file and >>>>>>>>>>> input files which I can place in my lib directory of hadoop and >>>>>>>>>>> test if >>>>>>>>>>> its working. And also the command to execute it. This should be >>>>>>>>>>> added in >>>>>>>>>>> the documentation as new comers can quickly setup giraph and >>>>>>>>>>> concentrate on >>>>>>>>>>> their project. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> It isn't asking for edge input. It says make sure you don't >>>>>>>>>>>> need it. A warning for the case you may have forgotten to give >>>>>>>>>>>> edge input >>>>>>>>>>>> when you really needed. >>>>>>>>>>>> >>>>>>>>>>>> The cause of your error is what I'm wondering nowadays. I'm >>>>>>>>>>>> having a similar problem. Currently I'm using a workaround: put >>>>>>>>>>>> all the >>>>>>>>>>>> jars (giraph-core and my module giraph-nutch) in the lib folder of >>>>>>>>>>>> hadoop. >>>>>>>>>>>> Then it works. But there should be a clean way of doing this. >>>>>>>>>>>> >>>>>>>>>>>> I should be able to say hadoop jar fat.jar ... >>>>>>>>>>>> >>>>>>>>>>>> Any help appreciated. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------ >>>>>>>>>>>> *Kimden: *"Vivek Sembium" <[email protected]> >>>>>>>>>>>> *Kime: *[email protected] >>>>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49 >>>>>>>>>>>> *Konu: *Re: Help needed for Running my own java programs in >>>>>>>>>>>> Giraph >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I tried with and without exporting hadoop classpath. I get the >>>>>>>>>>>> same error. >>>>>>>>>>>> >>>>>>>>>>>> Here's the command that I tried >>>>>>>>>>>> hadoop jar >>>>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar >>>>>>>>>>>> org.apache.giraph.GiraphRunner -libjars >>>>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar >>>>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute >>>>>>>>>>>> -vif >>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input >>>>>>>>>>>> format specified. Ensure your InputFormat does not require one. >>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output >>>>>>>>>>>> format specified. Ensure your OutputFormat does not require one. >>>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute >>>>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>>>>>>>> at >>>>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:634) >>>>>>>>>>>> at >>>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:212) >>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>>>> at >>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:314) >>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>>> at java.lang.Class.forName(Class.java:186) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>>>> at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>> Method) >>>>>>>>>>>> at >>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>>>> at >>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>>>> ... 26 more >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I gave the graph input as JSON file. Why does it ask for edge >>>>>>>>>>>> input file again? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Why isnt this working? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Vivek, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for the hasty response, I assumed the quick start guide >>>>>>>>>>>>> covered that (I guess it doesn't). What you need to do first is >>>>>>>>>>>>> put all of >>>>>>>>>>>>> your compiled code into a jar file. Then, you'll be utilizing >>>>>>>>>>>>> the hadoop >>>>>>>>>>>>> option -libjars. You also need to make sure that you set the >>>>>>>>>>>>> hadoop >>>>>>>>>>>>> classpath to include the giraph jar you are using as well as your >>>>>>>>>>>>> own jar. >>>>>>>>>>>>> I actually asked a similar question not too long ago; prior to >>>>>>>>>>>>> this, I was >>>>>>>>>>>>> just combining everything into one jar. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Your code should pretty much follow this format, which is >>>>>>>>>>>>> copied and paster from a previous response by Claudio: >>>>>>>>>>>>> >>>>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar" >>>>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D >>>>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass [ >>>>>>>>>>>>> GiraphRunner option e.g. -vip -vif etc. ]* >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thank You for your response. I could run all the example >>>>>>>>>>>>>> programs. I want to place my development code in a different >>>>>>>>>>>>>> directory >>>>>>>>>>>>>> other than the giraph directory. I copied the >>>>>>>>>>>>>> SimplePageRankComputation.java to a different directory, >>>>>>>>>>>>>> compiled it(with a >>>>>>>>>>>>>> different package name). But I couldnt replace the "org. >>>>>>>>>>>>>> apache.giraph.examples.SimpleShortestPathsComputation" with >>>>>>>>>>>>>> my class file. It gives me a ClassNotFoundException. Please >>>>>>>>>>>>>> Guide me so >>>>>>>>>>>>>> that I can run any class file with all required methods (GIven >>>>>>>>>>>>>> the location >>>>>>>>>>>>>> of that file). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> A "Quick Start" guide was recently added to the Giraph >>>>>>>>>>>>>>> website. Now learning Giraph is easier than ever! Here: >>>>>>>>>>>>>>> http://giraph.apache.org/quick_start.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am a new user in giraph. I could run all example programs >>>>>>>>>>>>>>>> in Giraph. Can someone please guide me on how to run my own >>>>>>>>>>>>>>>> java program in >>>>>>>>>>>>>>>> giraph (Assuming I have my input file in jason format in HDFS). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>>>>> University of Maryland >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>>> University of Maryland >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Claudio Martella >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claudio Martella >>>>>>>> [email protected] >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claudio Martella >>>>>> [email protected] >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Claudio Martella >>>> [email protected] >>>> >>> >>> >>> >>> -- >>> Kyle Orlando >>> Computer Engineering Major >>> University of Maryland >>> >> >> >> >> -- >> Claudio Martella >> [email protected] >> > > > > -- > Kyle Orlando > Computer Engineering Major > University of Maryland > -- Claudio Martella [email protected]
