Hi Jonathan, Can you be more specific about what problem you're running into?
SPARK-6869 fixed the issue of pyspark vs. assembly jar by shipping the pyspark archives separately to YARN. With that fix in place, pyspark doesn't need to get anything from the Spark assembly, so it has no problems running on YARN. I just downloaded spark-1.5.0-bin-hadoop2.6.tgz and tried that out, and pyspark works fine on YARN for me. On Fri, Aug 28, 2015 at 2:22 PM, Jonathan Bender <jonathan.ben...@gmail.com> wrote: > -1 for regression on PySpark + YARN support > > It seems like this JIRA https://issues.apache.org/jira/browse/SPARK-7733 > added a requirement for Java 7 in the build process. Due to some quirks > with the Java archive format changes between Java 6 and 7, using PySpark > with a YARN uberjar seems to break when compiled with anything after Java 6 > (see https://issues.apache.org/jira/browse/SPARK-1920 for reference). > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-0-RC2-tp13826p13890.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
