Re: Is the structure for a jar file for running Spark applications the same as that for Hadoop
On Mon, Sep 8, 2014 at 11:15 PM, Sean Owen so...@cloudera.com wrote: This structure is not specific to Hadoop, but in theory works in any JAR file. You can put JARs in JARs and refer to them with Class-Path entries in META-INF/MANIFEST.MF. Funny that you mention that, since someone internally asked the same question, and I spend some time looking at it. That's not actually how Class-Path works in the manifest. You can't have jars inside other jars; the Class-Path items reference things in the filesystem itself. So that solution doesn't work. It would be nice to add the feature Steve is talking about, though. -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is the structure for a jar file for running Spark applications the same as that for Hadoop
Hm, so it is: http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html I'm sure I've done this before though and thought is was this mechanism. It must be something custom. What's the Hadoop jar structure in question then? Is it something special like a WAR file? I confess I had never heard of this so thought this was about generic JAR stuff. Is the question about a lib dir in the Hadoop home dir? On Sep 10, 2014 11:34 PM, Marcelo Vanzin van...@cloudera.com wrote: On Mon, Sep 8, 2014 at 11:15 PM, Sean Owen so...@cloudera.com wrote: This structure is not specific to Hadoop, but in theory works in any JAR file. You can put JARs in JARs and refer to them with Class-Path entries in META-INF/MANIFEST.MF. Funny that you mention that, since someone internally asked the same question, and I spend some time looking at it. That's not actually how Class-Path works in the manifest. You can't have jars inside other jars; the Class-Path items reference things in the filesystem itself. So that solution doesn't work. It would be nice to add the feature Steve is talking about, though. -- Marcelo
Re: Is the structure for a jar file for running Spark applications the same as that for Hadoop
In modern projects there are a bazillion dependencies - when I use Hadoop I just put them in a lib directory in the jar - If I have a project that depends on 50 jars I need a way to deliver them to Spark - maybe wordcount can be written without dependencies but real projects need to deliver dependencies to the cluster On Wed, Sep 10, 2014 at 11:44 PM, Sean Owen so...@cloudera.com wrote: Hm, so it is: http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html I'm sure I've done this before though and thought is was this mechanism. It must be something custom. What's the Hadoop jar structure in question then? Is it something special like a WAR file? I confess I had never heard of this so thought this was about generic JAR stuff. Is the question about a lib dir in the Hadoop home dir? On Sep 10, 2014 11:34 PM, Marcelo Vanzin van...@cloudera.com wrote: On Mon, Sep 8, 2014 at 11:15 PM, Sean Owen so...@cloudera.com wrote: This structure is not specific to Hadoop, but in theory works in any JAR file. You can put JARs in JARs and refer to them with Class-Path entries in META-INF/MANIFEST.MF. Funny that you mention that, since someone internally asked the same question, and I spend some time looking at it. That's not actually how Class-Path works in the manifest. You can't have jars inside other jars; the Class-Path items reference things in the filesystem itself. So that solution doesn't work. It would be nice to add the feature Steve is talking about, though. -- Marcelo -- Steven M. Lewis PhD 4221 105th Ave NE Kirkland, WA 98033 206-384-1340 (cell) Skype lordjoe_com
Re: Is the structure for a jar file for running Spark applications the same as that for Hadoop
On Wed, Sep 10, 2014 at 3:44 PM, Sean Owen so...@cloudera.com wrote: What's the Hadoop jar structure in question then? Is it something special like a WAR file? I confess I had never heard of this so thought this was about generic JAR stuff. What I've been told (and Steve's e-mail alludes to) is that you can put jars inside a lib/ directory in your jar (so jars-within-a-jar), and the MR app classloader will automatically add those to your app's class path (by using a special class loader or exploding the jar or something, don't know the exact mechanism). -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is the structure for a jar file for running Spark applications the same as that for Hadoop
This structure is not specific to Hadoop, but in theory works in any JAR file. You can put JARs in JARs and refer to them with Class-Path entries in META-INF/MANIFEST.MF. It works but I have found it can cause trouble with programs that query the JARs on the classpath to find other classes. When they're in a JAR-within-a-JAR these approaches can fail. But, it's worth trying. You can always repackage all the JARs together, or include them individually on a classpath too, as alternatives that can also work. On Tue, Sep 9, 2014 at 2:56 AM, Steve Lewis lordjoe2...@gmail.com wrote: In a Hadoop jar there is a directory called lib and all non-provided third party jars go there and are included in the class path of the code. Do jars for Spark have the same structure - another way to ask the question is if I have code to execute Spark and a jar build for Hadoop can I simply use that jar? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
