Thank you, I will try distributed cache.
When I use distributed cache, the patches I have written will be unnecassary ? On 20 February 2012 03:44, David Garcia <dgar...@potomacfusion.com> wrote: > so, if that's the case, it's possible that the Tasktracker process doesn't > have the job on it's classpath. Although you have added the jar to "a" > classpath, I'm not certain that the Tasktracker will have it. There are > several ways to address this. 1.) you could bring Hadoop down, and then > adjust hadoop-env.sh to export the HADOOP_CLASSPATH environment variable to > include your jar. This variable is commented out by default. If you are > running in distributed mode, this means that you will have to copy this jar > to ever single machine...and probably change this script on every single > machine too...unless you are using something like condor (or puppet if > you're hard core serious), this is a serious pain...and for changing MR > jobs, totally overkill. My personal preference is to use the Distributed > cache, and copy your jar to a location in hdfs: > http://hadoop.apache.org/common/docs/r0.18.3/mapred_tutorial.html#DistributedCache > > hope this helps. > ________________________________________ > From: yavuz gokirmak [ygokir...@gmail.com] > Sent: Sunday, February 19, 2012 2:19 AM > To: giraph-user@incubator.apache.org > Subject: Re: how to use SimplePageRankVertex > > I am using pseudo distribudet cluster > > On 19 February 2012 02:00, David Garcia <dgar...@potomacfusion.com<mailto: > dgar...@potomacfusion.com>> wrote: > Are you submitting this job to a pseudo distributed cluster or a fully > distributed cluster? > > > Sent from my HTC Inspire™ 4G on AT&T > > ----- Reply message ----- > From: "yavuz gokirmak" <ygokir...@gmail.com<mailto:ygokir...@gmail.com>> > To: "giraph-user@incubator.apache.org<mailto: > giraph-user@incubator.apache.org>" <giraph-user@incubator.apache.org > <mailto:giraph-user@incubator.apache.org>> > Subject: how to use SimplePageRankVertex > Date: Sat, Feb 18, 2012 2:04 pm > > > > Thank you for advices, > > I have a few more questions. > > I have created a class named INTPageRankVertex which is similar to > SimplePageRankVertex and generated a jar holding only > INTPageRankVertex.java. > > Later, try to run with giraph command as below but get classpath errors: > > giraph INTPageRankVertex.jar org.test.INTPageRankVertex \ > -ip /user/hdfs/pagerankinput/graph.input \ > -op /user/hdfs/pagerankoutput/ \ > -w 1 \ > -if org.test.INTPageRankVertex.INTPageRankVertexInputFormat \ > -of org.test.INTPageRankVertex.INTPageRankVertexOutputFormat \ > > > First I get, > Exception in thread "main" java.lang.ClassNotFoundException: > org.test.INTPageRankVertex > > in bin/giraph user jar is added to classpath on line 58 > 58. CLASSPATH=${USER_JAR} > > but CLASSPATH is overwritten on line 87 > 87. CLASSPATH=`mvn dependency:build-classpath | grep -v "[INFO]"` > > changing line 87 as below solves my first problem. Does this patch is > valid? > 87. CLASSPATH=$CLASSPATH:`mvn dependency:build-classpath | grep -v > "[INFO]"` > > After changing line 87 I get a different classpath error: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/giraph/graph/LongDoubleFloatDoubleVertex > > And I solved this problem by adding below line > 113. CLASSPATH=$CLASSPATH:$JAR > > Does these patches are necessary or I am doing something wrong while > running my code.. > > best regards.. > > > On 18 February 2012 18:37, Avery Ching <ach...@apache.org<mailto: > ach...@apache.org>> wrote: > IntIntNullIntTextInputFormat in the examples package (extending > TextVertexInputFormat as David suggests) is very similar to what you need I > think, although the types might be different for your application. You can > start with that perhaps. > > Avery > > > On 2/18/12 7:48 AM, David Garcia wrote: > The easiest thing to do is to extend text vertex or/and textvertext input > format and/or the record reader. The record reader will give you the > vertices you want. Look at the record reader for textvertexinputformat. > It's an innerclass on this format class. > > Sent from my HTC Inspire™ 4G on AT&T > > ----- Reply message ----- > From: "yavuz gokirmak" <ygokir...@gmail.com><mailto:ygokir...@gmail.com> > To: "giraph-user@incubator.apache.org"<mailto: > giraph-user@incubator.apache.org> <giraph-user@incubator.apache.org > ><mailto:giraph-user@incubator.apache.org> > Subject: how to use SimplePageRankVertex > Date: Sat, Feb 18, 2012 9:08 am > > > > Hi, > > I am planning to use giraph for network analysis. First I am trying to > fully understand SimplePageRankVertex implementation and modify in order to > serve my needs. > > I have a question about example, > What is the expected input format for SimplePageRankVertex, I couldn't > understand the input format although SimplePageRankVertexReader class has > few lines. > > > My input file is contains of rows such as: > usera, userb > usera, userc > userc, usera > userb, userc > userc, userb > . > . > . > Each row represents a relation between two users, > "usera,userb" means that "usera is clicked userb's profile" > > Is it possible to make social network analysis over such kind of data > using giraph? > I will be glad if you can give advices.. > > thanks in advance > best regards > ygokirmak > > > >
