Hi Christoph,

Is your svn up to date?  I see bin/giraph in my svn checkout.

I am not sure that we have totally figured out the uber-fat-jar discussion. It's still evolving. Here's some quick possibilities you can try out though.

1)  Add your application to the Giraph code and compile the uber fat jar

2) mvn install your giraph jar into your local maven repo and then include it as a dependency for your applications. Then you can either include the application jar (uber jar approach) or you can put the jar in the library path for your hadoop cluster to reduce your application jar size.

Hope that helps!

Avery

On 12/23/11 5:11 AM, "Christoph Böhm" wrote:
Hi all,

first, I cannot find this node runner "bin/giraph" mentioned in GIRAPH-64 in 
the 0.7 trunk and, second, I'm not quite sure about the essence from that ueber-fat-jar 
discussion.
Is there a better/best practice way to run my implementation (vertex, io, of, 
etc) other then the ueber jar?

What am I missing?
Thanks.
Christoph




--- original msg ---
Gianmarco-
    GIRAPH-64 has been committed, so it should be much easier to treat
Giraph as a library and link your vertices against it.  Try it out and
see if it meets your needs.  If not, be sure to open a JIRA so we can
improve it.

Thanks,
Jakob


On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
<g...@apache.org>  wrote:
Nice!
It looks cleaner than my ant+ivy hack :)
Thanks for sharing,
--
Gianmarco


On Thu, Nov 10, 2011 at 16:18, Claudio Martella<claudio.marte...@gmail.com>
wrote:
Ok,

quite basically this is my pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
� � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
� � � �<modelVersion>4.0.0</modelVersion>
� � � �<groupId>org.acaro.graffiti</groupId>
� � � �<artifactId>graffiti</artifactId>
� � � �<packaging>jar</packaging>
� � � �<version>1.0-SNAPSHOT</version>
� � � �<name>graffiti</name>
� � � �<url>http://maven.apache.org</url>
� � � �<build>
� � � � � � � �<plugins>
� � � � � � � � � � � �<plugin>
� � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>

�<artifactId>maven-assembly-plugin</artifactId>
� � � � � � � � � � � � � � � �<version>2.2</version>
� � � � � � � � � � � � � � � �<configuration>

�<outputDirectory>target</outputDirectory>
� � � � � � � � � � � � � � � � � � � �<descriptorRefs>

�<descriptorRef>jar-with-dependencies</descriptorRef>
� � � � � � � � � � � � � � � � � � � �</descriptorRefs>
� � � � � � � � � � � � � � � �</configuration>
� � � � � � � � � � � � � � � �<executions>
� � � � � � � � � � � � � � � � � � � �<execution>
� � � � � � � � � � � � � � � � � � � � � � � �<id>make-assembly</id>
� � � � � � � � � � � � � � � � � � � � � � � �<phase>compile</phase>
� � � � � � � � � � � � � � � � � � � � � � � �<goals>
� � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>single</goal>
� � � � � � � � � � � � � � � � � � � � � � � �</goals>
� � � � � � � � � � � � � � � � � � � �</execution>
� � � � � � � � � � � � � � � �</executions>
� � � � � � � � � � � �</plugin>
� � � � � � � � � � � �<plugin>
� � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>

�<artifactId>maven-eclipse-plugin</artifactId>
� � � � � � � � � � � � � � � �<configuration>

�<downloadSources>true</downloadSources>

�<downloadJavadocs>true</downloadJavadocs>
� � � � � � � � � � � � � � � �</configuration>
� � � � � � � � � � � �</plugin>
� � � � � � � � � � � �<plugin>
� � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
� � � � � � � � � � � � � � � �<artifactId>maven-shade-plugin</artifactId>
� � � � � � � � � � � � � � � �<version>1.5</version>
� � � � � � � � � � � � � � � �<executions>
� � � � � � � � � � � � � � � � � � � �<execution>
� � � � � � � � � � � � � � � � � � � � � � � �<phase>package</phase>
� � � � � � � � � � � � � � � � � � � � � � � �<goals>
� � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>shade</goal>
� � � � � � � � � � � � � � � � � � � � � � � �</goals>
� � � � � � � � � � � � � � � � � � � � � � � �<configuration>
� � � � � � � � � � � � � � � � � � � � � � � �</configuration>
� � � � � � � � � � � � � � � � � � � �</execution>
� � � � � � � � � � � � � � � �</executions>
� � � � � � � � � � � �</plugin>
� � � � � � � �</plugins>
� � � �</build>
� � � �<dependencies>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>junit</groupId>
� � � � � � � � � � � �<artifactId>junit</artifactId>
� � � � � � � � � � � �<version>4.8.2</version>
� � � � � � � � � � � �<scope>test</scope>
� � � � � � � �</dependency>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>ch.qos.logback</groupId>
� � � � � � � � � � � �<artifactId>logback-classic</artifactId>
� � � � � � � � � � � �<version>0.9.27</version>
� � � � � � � �</dependency>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>com.google.guava</groupId>
� � � � � � � � � � � �<artifactId>guava</artifactId>
� � � � � � � � � � � �<version>r09</version>
� � � � � � � �</dependency>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>org.antlr</groupId>
� � � � � � � � � � � �<artifactId>antlr</artifactId>
� � � � � � � � � � � �<version>3.3</version>
� � � � � � � �</dependency>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>org.apache.giraph</groupId>
� � � � � � � � � � � �<artifactId>giraph</artifactId>
� � � � � � � � � � � �<version>0.70</version>
� � � � � � � �</dependency>
� � � � � � � �<dependency>
� � � � � � � � � � � �<groupId>org.apache.hadoop</groupId>
� � � � � � � � � � � �<artifactId>hadoop-core</artifactId>
� � � � � � � � � � � �<version>0.20.204.0</version>
� � � � � � � � � � � �<scope>provided</scope>
� � � � � � � �</dependency>
� � � �</dependencies>
</project>


You set your project dependecies as usual, set hadoop-core (you don't
really need it in the job jar) as<provided>  and then use the
maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
package goal it will upack your dependencies and re-pack them along
with your project classes in an uberjar.

maven-shade allows you to set excludes in case of conflicts.

Easy, automatic and clean.


Hope this helps,
Claudio

On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<ach...@apache.org>  wrote:
Would be great if you can document what you did. =)

Thanks,

Avery

On 11/8/11 3:13 PM, Claudio Martella wrote:
Sorry guys, may bad.

Was calling job.waitForCompletion() directly. I've been coding
standard mapreduce whole weekend...

Anyway I got a solution for clean packaging of your own application
over giraph, and that is exactly using maven-shade-plugin. it will
prepare the uberjar for you.

On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
<claudio.marte...@gmail.com>  �wrote:
Hello list,

I'm actually having troubles as well to get my application running.

I've give a shot to maven-shade plugin which unpacks my dependencies
and packs them all together with my classes in a new jar.

I attach the hierarchy of the jar so that somebody can maybe spot
what's missing, because i can't get it working. I get an identity
map-reduce job with jobconf complaining about no job jar being set.

Any idea?

On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ach...@apache.org>  �wrote:
Hi Gianmarco,

You're right, most of us (to my knowledge) have been using Giraph
with
an
uberjar as you've put it. �However, Jakob has been doing some work to
make
this easier. �See the below issue:

https://issues.apache.org/jira/browse/GIRAPH-64

If you can suggest a better approach, please add to the issue or
create
a
new one if appropriate.

Thanks,

Avery

On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
Hi community,

I was wondering what is the current best practice to package an
application in a jar for deployment.
I tried the 'hadoop way' by putting giraph-*.jar in the /lib
directory
of
my jar, and using the -libjars option but none of them worked. It
looks
like
the backend classloader is doing some mess and it doesn't find my
own
classes in the jar.

I resorted to uncompressing the giraph-*.jar and repackaging my
classes
with it, all at the same level (an uber-fat jar), but even though it
works
it doesn't sound like the right approach.

Any suggestions?

Thanks,
--
Gianmarco



Reply via email to