Hi, I think you should be able to handle this with 0.6 release using Distributed cache. Did the new release work for you?
-Suraj On Wed, Nov 14, 2012 at 9:04 PM, Edward J. Yoon <[email protected]>wrote: > Can you provide me your tasklogs? > > > I am writing my own VertexInputReader and need a third party library to > > parse the input. > > If you added your jars to {$HAMA_HOME}/lib folder (all hosts), you can > check whether your jars are included as following command. > > [oracle@bda01 hama-0.6.0]$ ps -ef | grep Groom > > oracle 29928 1 23 17:25 ? 00:00:01 > /usr/java/jdk1.6.0_29/bin/java -Xmx2000m > -Dhama.log.dir=/home/oracle/hama-0.6.0/bin/../logs > -Dhama.log.file=hama-oracle-groom-bda01.oracle.com.log > -Dhama.home.dir=/home/oracle/hama-0.6.0/bin/.. -Dhama.id.str=oracle > -Dhama.root.logger=INFO,DRFA -Dhama.policy.file=hama-policy.xml > -classpath > /home/oracle/hama-0.6.0/bin/../conf:/usr/java/jdk1.6.0_29/lib/tools.jar:/home/oracle/hama-0.6.0/bin/../hama-core-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-examples-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-graph-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-ml-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-launcher-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-ipc-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-cli-1.2.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-configuration-1.7.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-httpclient-3.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-lang-2.6.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-logging-1.1.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-math3-3.0.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-10.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-r09-jarjar.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-core-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-test-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-core-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-mapper-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-annotations-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-util-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/junit-4.8.1.jar:/home/oracle/hama-0.6.0/bin/../lib/log4j-1.2.16.jar:/home/oracle/hama-0.6.0/bin/../lib/netty-3.2.6.Final.jar:/home/oracle/hama-0.6.0/bin/../lib/servlet-api-6.0.32.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-api-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-log4j12-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/snappy-java-1.0.4.1.jar:/home/oracle/hama-0.6.0/bin/../lib/zookeeper-3.3.3.jar > org.apache.hama.GroomServerRunner bda01-adm > oracle 30029 29694 0 17:25 pts/0 00:00:00 grep Groom > > > On Thu, Nov 15, 2012 at 10:18 AM, Tu, Min <[email protected]> wrote: > > Hi Edward and Suraj, > > > > Thank you very much for spending time answer my questions. Really > > appreciated. > > > > I have tried to put my library to $HAMA_HOME/lib folder in both BSP and > > Groom hosts. > > > > The BSP server pick it up fine but Groom server still starts without that > > package in the class path. > > > > Any thing I did wrong? > > > > Also Is there another way to ship the 3rd party jar without put those > into > > the $HAMA_HOME/lib? Like Distributed cache in hadoop? > > > > Here are the hama command and partial code : > > > > hama jar ./POC-0.0.1-SNAPSHOT.jar > > com.paypal.hadoop.platform.graphjob.SimpleRiskPropagation > > /user/mitu/riskData/DW_PAYMENT_SENT/preprocess/output/2nd > > /user/mitu/riskData/DW_PAYMENT_SENT/propagate/output/2nd > > > > > > --------------------- Main function in the SimpleRiskPropagation.java > ----- > > public static void main(String... args) throws IOException, > > InterruptedException, ClassNotFoundException { > > > > if (args.length < 2) { > > printUsage(); > > } > > > > HamaConfiguration conf = new HamaConfiguration(new > Configuration()); > > GraphJob job = new GraphJob(conf, > SimpleRiskPropagation.class); > > > > // set the defaults > > //job.setMaxIteration(30); > > job.setMaxIteration(3); > > job.setNumBspTask(2); > > > > if (args.length >= 4) > > job.setNumBspTask(Integer.parseInt(args[3])); > > if (args.length >= 3) > > job.setMaxIteration(Integer.parseInt(args[2])); > > > > job.setJobName("SimpleRiskPropagation"); > > job.setInputPath(new Path(args[0])); > > job.setOutputPath(new Path(args[1])); > > > > job.setVertexClass(SimpleRiskPropagationVertex.class); > > job.setVertexIDClass(Text.class); > > job.setVertexValueClass(Pairs.class); > > job.setEdgeValueClass(SimpleTransaction.class); > > > > job.setInputKeyClass(LongWritable.class); > > job.setInputValueClass(Text.class); > > job.setInputFormat(TextInputFormat.class); > > > > > > /* > > * this reader class will be set in GraphJob conf with > key: > > VERTEX_GRAPH_INPUT_READER > > * and used by class GraphJobRunner 's function > loadVertices() > > */ > > > job.setVertexInputReaderClass(SimpleRiskPropagationVertexReader.class); > > > > > > job.setPartitioner(HashPartitioner.class); > > job.setOutputFormat(TextOutputFormat.class); > > job.setOutputKeyClass(Text.class); > > job.setOutputValueClass(Text.class); > > > > long startTime = System.currentTimeMillis(); > > if (job.waitForCompletion(true)) { > > System.out.println("Job Finished in " > > + (System.currentTimeMillis() - > startTime) / 1000.0 + " seconds"); > > } > > } > > --------------------- Main function in the SimpleRiskPropagation.java > ----- > > > > > > > > On 11/13/12 6:13 AM, "Edward J. Yoon" <[email protected]> wrote: > > > >>Did you subscribe? > >> > >>---------- Forwarded message ---------- > >>From: Edward J. Yoon <[email protected]> > >>Date: Tue, Nov 13, 2012 at 7:42 PM > >>Subject: Re: Load user library to Hama > >>To: [email protected] > >> > >> > >>Hi Tu, Min > >> > >>You can solve the problem by copying your 3rd party library to > >>{$HAMA}/lib folder. > >> > >>On Tue, Nov 13, 2012 at 7:32 PM, Suraj Menon <[email protected]> > >>wrote: > >>> Hi, > >>> > >>> Can you give us the command you are trying. Also, you can try the > >>> distributed cache feature of Hadoop for the same. > >>> > >>> Thanks, > >>> Suraj > >>> > >>> > >>> On Mon, Nov 12, 2012 at 9:03 PM, Tu, Min <[email protected]> wrote: > >>> > >>>> Hi, > >>>> > >>>> I am new to Hama and got a question for using it. > >>>> > >>>> I am writing my own VertexInputReader and need a third party library > to > >>>> parse the input. How should I ship the library jar to the GroomServer > >>>>in > >>>> Hama. I have tried to use the libjars option but did not work: got > >>>> ClassNotFoundException in the VertexInputReader part. > >>>> > >>>> > >>>> Thanks a lot > >>>> > >>>> Min > >>>> > >> > >> > >> > >>-- > >>Best Regards, Edward J. Yoon > >>@eddieyoon > >> > >> > >>-- > >>Best Regards, Edward J. Yoon > >>@eddieyoon > > > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon >
