[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394652#comment-15394652
]
Sean Owen commented on SPARK-16725:
-----------------------------------
Again there are two Guavas here: the unshaded one for Hadoop et al, and the
shaded one for Spark. The unshaded one is 14. If you update it, other things
break.
(In 1.6.x, there are actually unshaded Guava Optional in the assembly for
historical reasons, but I don't know if that would trigger the warning.)
Spark requires Hadoop classes one way or the other. You certainly have them
somewhere or nothing will work. You can build Spark to build in Hadoop classes
or have them provided at runtime. Maybe you need the latter. If the runtime
doesn't provide the dependency, how can any app work without its dependency
bundled?
Yes, it's likely that the ideal resolution here is for spark-cassandra to shade
Guava to be safe, if it's bundling unshaded Guava. I'm not suggesting it's your
change to make, but, not Spark's either.
Shading means moving the dependency to a different namespace, not sure hat you
mean. It's probably not realistic to get rid of Guava; it's useful. Even if you
did the problem is that dependencies may not. I don't think it's random;
versions are as high as they can be without apparently breaking compatibility
with how other commonly-paired dependencies work. That is, there's a reason
it's 14 and not 15.
> Migrate Guava to 16+?
> ---------------------
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 2.0.1
> Reporter: Min Wei
> Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> - val cookie = HashCodes.fromBytes(secret).toString()
> + val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
> } else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
> // A general, soft-reference map for metadata needed during HadoopRDD
> split computation
> // (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
> private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> - <version>14.0.1</version>
> - <scope>provided</scope>
> + <version>19.0</version>
> </dependency>
> <!-- End of shaded deps -->
> <dependency>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
