[
https://issues.apache.org/jira/browse/SPARK-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-3234:
------------------------------
Description:
In {{make-distribution.sh}}, {{SPARK_HADOOP_VERSION}} is determined by
{{\-\-hadoop}}, and {{SPARK_HIVE}} is determined by {{\-\-with-hive}}, but
these two command line options are deprecated by
`-Dhadoop.version`/`-Phadoop-??` and `-Phive` respectively. The result is that:
#. we may end up with something like {{spark-1.1.0-SNAPSHOT-bin-.tgz}} unless
`--name` is specified explicitly
#. Datanucleus jars will not be included in the distribution folder/tarball
unless we set {{SPARK_HIVE=true}} explicitly
A possible fix is to check {{hadoop.version}} and {{project.activeProfiles}}
with {{mvn help:evaluate}}, similar to what we do now to determine {{VERSION}}.
was:
{{SPARK_HADOOP_VERSION}} has already been deprecated, but
{{make-distribution.sh}} uses it as part of the distribution tarball name. As a
result, we end up with something like {{spark-1.1.0-SNAPSHOT-bin-.tgz}} because
{{SPARK_HADOOP_VERSION}} is empty.
A possible fix is to add the antrun plugin into the Maven build and run Maven
to print {{$hadoop.version}}. Instructions can be found in [this
post|http://www.avajava.com/tutorials/lessons/how-do-i-display-the-value-of-a-property.html].
> SPARK_HADOOP_VERSION and SPARK_HIVE depend on deprecated make-distribution.sh
> command line options
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-3234
> URL: https://issues.apache.org/jira/browse/SPARK-3234
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.0.2
> Reporter: Cheng Lian
> Priority: Minor
>
> In {{make-distribution.sh}}, {{SPARK_HADOOP_VERSION}} is determined by
> {{\-\-hadoop}}, and {{SPARK_HIVE}} is determined by {{\-\-with-hive}}, but
> these two command line options are deprecated by
> `-Dhadoop.version`/`-Phadoop-??` and `-Phive` respectively. The result is
> that:
> #. we may end up with something like {{spark-1.1.0-SNAPSHOT-bin-.tgz}} unless
> `--name` is specified explicitly
> #. Datanucleus jars will not be included in the distribution folder/tarball
> unless we set {{SPARK_HIVE=true}} explicitly
> A possible fix is to check {{hadoop.version}} and {{project.activeProfiles}}
> with {{mvn help:evaluate}}, similar to what we do now to determine
> {{VERSION}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
