Thanks Avery! I don't like the uber-fat-jar since I work remote and doploy on the cluster via a not-super-fast-connection. The first approach I actually tried was to "put the jar in the library path" which didn't work. Think that I got to deal with the fat jar for a while. Christoph
-------- Original-Nachricht -------- > Datum: Fri, 23 Dec 2011 09:19:37 -0800 > Von: Avery Ching <ach...@apache.org> > An: giraph-user@incubator.apache.org > Betreff: Re: Packaging a Giraph application in a jar > Hi Christoph, > > Is your svn up to date? I see bin/giraph in my svn checkout. > > I am not sure that we have totally figured out the uber-fat-jar > discussion. It's still evolving. Here's some quick possibilities you > can try out though. > > 1) Add your application to the Giraph code and compile the uber fat jar > > 2) mvn install your giraph jar into your local maven repo and then > include it as a dependency for your applications. Then you can either > include the application jar (uber jar approach) or you can put the jar > in the library path for your hadoop cluster to reduce your application > jar size. > > Hope that helps! > > Avery > > On 12/23/11 5:11 AM, "Christoph Böhm" wrote: > > Hi all, > > > > first, I cannot find this node runner "bin/giraph" mentioned in > GIRAPH-64 in the 0.7 trunk and, second, I'm not quite sure about the essence > from > that ueber-fat-jar discussion. > > Is there a better/best practice way to run my implementation (vertex, > io, of, etc) other then the ueber jar? > > > > What am I missing? > > Thanks. > > Christoph > > > > > > > > > > --- original msg --- > > Gianmarco- > > GIRAPH-64 has been committed, so it should be much easier to treat > > Giraph as a library and link your vertices against it. Try it out and > > see if it meets your needs. If not, be sure to open a JIRA so we can > > improve it. > > > > Thanks, > > Jakob > > > > > > On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales > > <g...@apache.org> wrote: > >> Nice! > >> It looks cleaner than my ant+ivy hack :) > >> Thanks for sharing, > >> -- > >> Gianmarco > >> > >> > >> On Thu, Nov 10, 2011 at 16:18, Claudio > Martella<claudio.marte...@gmail.com> > >> wrote: > >>> Ok, > >>> > >>> quite basically this is my pom.xml: > >>> > >>> <project xmlns="http://maven.apache.org/POM/4.0.0" > >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > >>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > >>> http://maven.apache.org/maven-v4_0_0.xsd"> > >>> � � � �<modelVersion>4.0.0</modelVersion> > >>> � � � �<groupId>org.acaro.graffiti</groupId> > >>> � � � �<artifactId>graffiti</artifactId> > >>> � � � �<packaging>jar</packaging> > >>> � � � �<version>1.0-SNAPSHOT</version> > >>> � � � �<name>graffiti</name> > >>> � � � �<url>http://maven.apache.org</url> > >>> � � � �<build> > >>> � � � � � � � �<plugins> > >>> � � � � � � � � � � � �<plugin> > >>> � � � � � � � � � � � � � � � > �<groupId>org.apache.maven.plugins</groupId> > >>> > >>> �<artifactId>maven-assembly-plugin</artifactId> > >>> � � � � � � � � � � � � � � � > �<version>2.2</version> > >>> � � � � � � � � � � � � � � � > �<configuration> > >>> > >>> �<outputDirectory>target</outputDirectory> > >>> � � � � � � � � � � � � � � � � � > � � �<descriptorRefs> > >>> > >>> �<descriptorRef>jar-with-dependencies</descriptorRef> > >>> � � � � � � � � � � � � � � � � � > � � �</descriptorRefs> > >>> � � � � � � � � � � � � � � � > �</configuration> > >>> � � � � � � � � � � � � � � � > �<executions> > >>> � � � � � � � � � � � � � � � � � > � � �<execution> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<id>make-assembly</id> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<phase>compile</phase> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<goals> > >>> � � � � � � � � � � � � � � � � � > � � � � � � � � � � �<goal>single</goal> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �</goals> > >>> � � � � � � � � � � � � � � � � � > � � �</execution> > >>> � � � � � � � � � � � � � � � > �</executions> > >>> � � � � � � � � � � � �</plugin> > >>> � � � � � � � � � � � �<plugin> > >>> � � � � � � � � � � � � � � � > �<groupId>org.apache.maven.plugins</groupId> > >>> > >>> �<artifactId>maven-eclipse-plugin</artifactId> > >>> � � � � � � � � � � � � � � � > �<configuration> > >>> > >>> �<downloadSources>true</downloadSources> > >>> > >>> �<downloadJavadocs>true</downloadJavadocs> > >>> � � � � � � � � � � � � � � � > �</configuration> > >>> � � � � � � � � � � � �</plugin> > >>> � � � � � � � � � � � �<plugin> > >>> � � � � � � � � � � � � � � � > �<groupId>org.apache.maven.plugins</groupId> > >>> � � � � � � � � � � � � � � � > �<artifactId>maven-shade-plugin</artifactId> > >>> � � � � � � � � � � � � � � � > �<version>1.5</version> > >>> � � � � � � � � � � � � � � � > �<executions> > >>> � � � � � � � � � � � � � � � � � > � � �<execution> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<phase>package</phase> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<goals> > >>> � � � � � � � � � � � � � � � � � > � � � � � � � � � � �<goal>shade</goal> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �</goals> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �<configuration> > >>> � � � � � � � � � � � � � � � � � > � � � � � � �</configuration> > >>> � � � � � � � � � � � � � � � � � > � � �</execution> > >>> � � � � � � � � � � � � � � � > �</executions> > >>> � � � � � � � � � � � �</plugin> > >>> � � � � � � � �</plugins> > >>> � � � �</build> > >>> � � � �<dependencies> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>junit</groupId> > >>> � � � � � � � � � � � > �<artifactId>junit</artifactId> > >>> � � � � � � � � � � � > �<version>4.8.2</version> > >>> � � � � � � � � � � � �<scope>test</scope> > >>> � � � � � � � �</dependency> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>ch.qos.logback</groupId> > >>> � � � � � � � � � � � > �<artifactId>logback-classic</artifactId> > >>> � � � � � � � � � � � > �<version>0.9.27</version> > >>> � � � � � � � �</dependency> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>com.google.guava</groupId> > >>> � � � � � � � � � � � > �<artifactId>guava</artifactId> > >>> � � � � � � � � � � � �<version>r09</version> > >>> � � � � � � � �</dependency> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>org.antlr</groupId> > >>> � � � � � � � � � � � > �<artifactId>antlr</artifactId> > >>> � � � � � � � � � � � �<version>3.3</version> > >>> � � � � � � � �</dependency> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>org.apache.giraph</groupId> > >>> � � � � � � � � � � � > �<artifactId>giraph</artifactId> > >>> � � � � � � � � � � � �<version>0.70</version> > >>> � � � � � � � �</dependency> > >>> � � � � � � � �<dependency> > >>> � � � � � � � � � � � > �<groupId>org.apache.hadoop</groupId> > >>> � � � � � � � � � � � > �<artifactId>hadoop-core</artifactId> > >>> � � � � � � � � � � � > �<version>0.20.204.0</version> > >>> � � � � � � � � � � � �<scope>provided</scope> > >>> � � � � � � � �</dependency> > >>> � � � �</dependencies> > >>> </project> > >>> > >>> > >>> You set your project dependecies as usual, set hadoop-core (you don't > >>> really need it in the job jar) as<provided> and then use the > >>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At > >>> package goal it will upack your dependencies and re-pack them along > >>> with your project classes in an uberjar. > >>> > >>> maven-shade allows you to set excludes in case of conflicts. > >>> > >>> Easy, automatic and clean. > >>> > >>> > >>> Hope this helps, > >>> Claudio > >>> > >>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<ach...@apache.org> > wrote: > >>>> Would be great if you can document what you did. =) > >>>> > >>>> Thanks, > >>>> > >>>> Avery > >>>> > >>>> On 11/8/11 3:13 PM, Claudio Martella wrote: > >>>>> Sorry guys, may bad. > >>>>> > >>>>> Was calling job.waitForCompletion() directly. I've been coding > >>>>> standard mapreduce whole weekend... > >>>>> > >>>>> Anyway I got a solution for clean packaging of your own application > >>>>> over giraph, and that is exactly using maven-shade-plugin. it will > >>>>> prepare the uberjar for you. > >>>>> > >>>>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella > >>>>> <claudio.marte...@gmail.com> �wrote: > >>>>>> Hello list, > >>>>>> > >>>>>> I'm actually having troubles as well to get my application running. > >>>>>> > >>>>>> I've give a shot to maven-shade plugin which unpacks my > dependencies > >>>>>> and packs them all together with my classes in a new jar. > >>>>>> > >>>>>> I attach the hierarchy of the jar so that somebody can maybe spot > >>>>>> what's missing, because i can't get it working. I get an identity > >>>>>> map-reduce job with jobconf complaining about no job jar being set. > >>>>>> > >>>>>> Any idea? > >>>>>> > >>>>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ach...@apache.org> > �wrote: > >>>>>>> Hi Gianmarco, > >>>>>>> > >>>>>>> You're right, most of us (to my knowledge) have been using Giraph > >>>>>>> with > >>>>>>> an > >>>>>>> uberjar as you've put it. �However, Jakob has been doing some > work to > >>>>>>> make > >>>>>>> this easier. �See the below issue: > >>>>>>> > >>>>>>> https://issues.apache.org/jira/browse/GIRAPH-64 > >>>>>>> > >>>>>>> If you can suggest a better approach, please add to the issue or > >>>>>>> create > >>>>>>> a > >>>>>>> new one if appropriate. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Avery > >>>>>>> > >>>>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote: > >>>>>>>> Hi community, > >>>>>>>> > >>>>>>>> I was wondering what is the current best practice to package an > >>>>>>>> application in a jar for deployment. > >>>>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib > >>>>>>>> directory > >>>>>>>> of > >>>>>>>> my jar, and using the -libjars option but none of them worked. It > >>>>>>>> looks > >>>>>>>> like > >>>>>>>> the backend classloader is doing some mess and it doesn't find my > >>>>>>>> own > >>>>>>>> classes in the jar. > >>>>>>>> > >>>>>>>> I resorted to uncompressing the giraph-*.jar and repackaging my > >>>>>>>> classes > >>>>>>>> with it, all at the same level (an uber-fat jar), but even though > it > >>>>>>>> works > >>>>>>>> it doesn't sound like the right approach. > >>>>>>>> > >>>>>>>> Any suggestions? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> -- > >>>>>>>> Gianmarco > >>>>>>>> > >>>>>>>> >