Barry Becker created SPARK-19245:
------------------------------------
Summary: Cannot build spark-assembly jar
Key: SPARK-19245
URL: https://issues.apache.org/jira/browse/SPARK-19245
Project: Spark
Issue Type: Documentation
Components: Build
Affects Versions: 2.1.0
Reporter: Barry Becker
Priority: Minor
This may just be an issue with documentation, but I am not sure.
In previous versions of spark (spark 2.0.2 and earlier) one could execute a
command like
build/mvn -Pmesos -Phadoop-2.6 -Dhadoop.version=2.6.5 -Dscala-2.11 -DskipTests
clean package
and have it produce an assembly jar called something like
spark-assembly-x.y.z-hadoop2.6.5.jar. This no longer seems possible with spark
2.1.0.
There is a -bigtop-dist option (not documented on the main 2.1 build page) that
allows you to
create a spark-assembly_2.11-2.1.0-dist.tar.gzip in spark-2.1.0/assembly/target.
If I unpack it, it contains a lot of jars and a README that states:
{quote}
This is an assembly module for Spark project.
It creates a single tar.gz file that includes all needed dependency of the
project except for org.apache.hadoop.* jars that are supposed to be available
from the deployed Hadoop cluster.
This module is off by default. To activate it specify the profile in the
command line
-Pbigtop-dist
If you need to build an assembly for a different version of Hadoop the
hadoop-version system property needs to be set as in this example:
-Dhadoop.version=2.0.6-alpha
{quote}
My question is, how can I produce a single
mineset-assembly-x.y.y-hadoopA.B.C.jar? If that it no longer possible, how are
we supposed to use the bigtop-dist.tar.gzip file? Do we copy it to the same lib
directory and unpack it there? In any case, some clarification on the build
page (http://spark.apache.org/docs/latest/building-spark.html) would help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
