[
https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502675#comment-14502675
]
Steve Loughran commented on SPARK-7009:
---------------------------------------
Looking at the [openJDK
issue|https://bugs.openjdk.java.net/browse/JDK-4828461], Java6 appears to be
generating a header/footer that stops at 64K, and doesn't bother reading that
header when enumerating zip file. Java 7 (presumably) handles reads the same
way, but uses zip64 to generate the artifacts. Ant can be told not to generate
zip64 files, but it does zip16 "properly", rejecting source filesets that are
too large
There isn't an obvious/immediate solution for this on Java7+; except to extend
Ant to generate the same hacked zip files, then wait for that to trickle into
the maven ant-run plugin, which would be about 3+ months after ant 1.9.x ships.
That's a long term project, though something to consider starting now, to get
the feature later in 2015
> Build assembly JAR via ant to avoid zip64 problems
> --------------------------------------------------
>
> Key: SPARK-7009
> URL: https://issues.apache.org/jira/browse/SPARK-7009
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.0
> Environment: Java 7+
> Reporter: Steve Loughran
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a
> format incompatible with Java and pyspark.
> Provided the total number of .class files+resources is <64K, ant can be used
> to make the final JAR instead, perhaps by unzipping the maven-generated JAR
> then rezipping it with zip64=never, before publishing the artifact via maven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
