[
https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502740#comment-14502740
]
Steve Loughran commented on SPARK-7009:
---------------------------------------
The problem we are seeing is pyspark related; (SPARK-1753 ?) it doesn't like
Java7-built artifacts, and appears to handle that ugly hack in late java6 JVMs.
Java7 is/should be the runtime to build things (even that's going to be hard to
D/L soon), and as more things go to java7+ binaries (e.g. Hadoop 2.7+), the
option of building on Java6 goes away.
If pyspark doesn't like the java7 assembly JAR, then the options are limited
# fix python's zip stuff
# fix ant to replicate the java 6 hack
# fix openjdk
# something ugly involving a build with java7, unjar and then a rejar with java6
short term, option #4 is the only one that works; that can live outside the mvn
build itself. #2 is doable; I could probably get that patch in by the end of
the week, then after ant 1.9.5 ships tweak the mvn:antrun plugin to use the
later dependency. openjdk is probably too hard, leaving pyspark the final fix
point.
> Build assembly JAR via ant to avoid zip64 problems
> --------------------------------------------------
>
> Key: SPARK-7009
> URL: https://issues.apache.org/jira/browse/SPARK-7009
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.0
> Environment: Java 7+
> Reporter: Steve Loughran
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a
> format incompatible with Java and pyspark.
> Provided the total number of .class files+resources is <64K, ant can be used
> to make the final JAR instead, perhaps by unzipping the maven-generated JAR
> then rezipping it with zip64=never, before publishing the artifact via maven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
