[
https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589913#comment-14589913
]
Harry Brundage edited comment on SPARK-7009 at 6/17/15 3:22 PM:
----------------------------------------------------------------
[~sowen] I wrote a script to test that we can all run to prove if it's
importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and
hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1
hadoop-2.6 artifact works fine for me on my machine. Script attached.
If this is the case for others, then we're in a spot of trouble, no?
was (Author: airhorns):
[~sowen] I wrote a script to test that we can all run to prove if it's
importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and
hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1
hadoop-2.6 artifact works fine for me on my machine. Script attached.
If this is the case for other, then we're in a spot of trouble, no?
> Build assembly JAR via ant to avoid zip64 problems
> --------------------------------------------------
>
> Key: SPARK-7009
> URL: https://issues.apache.org/jira/browse/SPARK-7009
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.0
> Environment: Java 7+
> Reporter: Steve Loughran
> Attachments: check_spark_python.sh
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a
> format incompatible with Java and pyspark.
> Provided the total number of .class files+resources is <64K, ant can be used
> to make the final JAR instead, perhaps by unzipping the maven-generated JAR
> then rezipping it with zip64=never, before publishing the artifact via maven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
