This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3909223 [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it 3909223 is described below commit 39092236819da097e9c8a3b2fa975105f08ae5b9 Author: Sean Owen <sean.o...@databricks.com> AuthorDate: Tue Mar 5 08:26:30 2019 -0600 [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it ## What changes were proposed in this pull request? Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency. Packaging Spark makes the app much bigger by tens of megabytes. It can also bring in conflicting dependencies that wouldn't otherwise be a problem. https://issues.apache.org/jira/browse/SPARK-26146 was what reminded me of this. ## How was this patch tested? Doc build Closes #23938 from srowen/Provided. Authored-by: Sean Owen <sean.o...@databricks.com> Signed-off-by: Sean Owen <sean.o...@databricks.com> --- docs/cloud-integration.md | 1 + docs/quick-start.md | 1 + docs/streaming-programming-guide.md | 3 ++- 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/cloud-integration.md b/docs/cloud-integration.md index 5368e13..a014d6f 100644 --- a/docs/cloud-integration.md +++ b/docs/cloud-integration.md @@ -87,6 +87,7 @@ is set to the chosen version of Spark: <groupId>org.apache.spark</groupId> <artifactId>hadoop-cloud_{{site.SCALA_BINARY_VERSION}}</artifactId> <version>${spark.version}</version> + <scope>provided</scope> </dependency> ... </dependencyManagement> diff --git a/docs/quick-start.md b/docs/quick-start.md index 28186c1..8bf0d6e 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -341,6 +341,7 @@ Note that Spark artifacts are tagged with a Scala version. <groupId>org.apache.spark</groupId> <artifactId>spark-sql_{{site.SCALA_BINARY_VERSION}}</artifactId> <version>{{site.SPARK_VERSION}}</version> + <scope>provided</scope> </dependency> </dependencies> </project> diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md index 94c6120..30a0160 100644 --- a/docs/streaming-programming-guide.md +++ b/docs/streaming-programming-guide.md @@ -385,11 +385,12 @@ Similar to Spark, Spark Streaming is available through Maven Central. To write y <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_{{site.SCALA_BINARY_VERSION}}</artifactId> <version>{{site.SPARK_VERSION}}</version> + <scope>provided</scope> </dependency> </div> <div data-lang="SBT" markdown="1"> - libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}" + libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}" % "provided" </div> </div> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org