On Wed, Sep 10, 2014 at 3:48 PM, Steve Lewis wrote:
> In modern projects there are a bazillion dependencies - when I use Hadoop I
> just put them in a lib directory in the jar - If I have a project that
> depends on 50 jars I need a way to deliver them to Spark - maybe wordcount
> can be written w
On Wed, Sep 10, 2014 at 3:44 PM, Sean Owen wrote:
> What's the Hadoop jar structure in question then? Is it something special
> like a WAR file? I confess I had never heard of this so thought this was
> about generic JAR stuff.
What I've been told (and Steve's e-mail alludes to) is that you can
p
In modern projects there are a bazillion dependencies - when I use Hadoop I
just put them in a lib directory in the jar - If I have a project that
depends on 50 jars I need a way to deliver them to Spark - maybe wordcount
can be written without dependencies but real projects need to deliver
depende
Hm, so it is:
http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
I'm sure I've done this before though and thought is was this mechanism. It
must be something custom.
What's the Hadoop jar structure in question then? Is it something special
like a WAR file? I confess I had never h
On Mon, Sep 8, 2014 at 11:15 PM, Sean Owen wrote:
> This structure is not specific to Hadoop, but in theory works in any
> JAR file. You can put JARs in JARs and refer to them with Class-Path
> entries in META-INF/MANIFEST.MF.
Funny that you mention that, since someone internally asked the same
q
This structure is not specific to Hadoop, but in theory works in any
JAR file. You can put JARs in JARs and refer to them with Class-Path
entries in META-INF/MANIFEST.MF.
It works but I have found it can cause trouble with programs that
query the JARs on the classpath to find other classes. When t
In a Hadoop jar there is a directory called lib and all non-provided third
party jars go there and are included in the class path of the code. Do jars
for Spark have the same structure - another way to ask the question is if I
have code to execute Spark and a jar build for Hadoop can I simply use