[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?
[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394768#comment-15394768
]
Min Wei commented on SPARK-16725:
-
Still worth to use a newer version of Guava. Looks like the issue of upgrading
to Guava v16+ is postponed to Hadoop 3.0 at least.
https://issues.apache.org/jira/browse/HADOOP-11319
Hopefully Guava devs. will be more disciplined with its API compatibility.
There seems to be quite many JIRAs on Guava.
> Migrate Guava to 16+?
> -
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 2.0.1
>Reporter: Min Wei
>Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> -val cookie = HashCodes.fromBytes(secret).toString()
> +val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
>} else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
>// A general, soft-reference map for metadata needed during HadoopRDD
> split computation
>// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
>private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
>
> com.google.guava
> guava
> -14.0.1
> -provided
> +19.0
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?
[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394622#comment-15394622
]
Min Wei commented on SPARK-16725:
-
I would love to see Spark successful as a platform, and not caught up in some
versioning mess down the road.
>Spark shades Guava and therefore doesn't leak it.
This does not seem to be true. In my case, I only used spark-shell and
spark-cassandra in the standalone environment, no Hadoop bits (at least not
explicitly).
>Spark depends on Hadoop, Hadoop depends on unshaded Guava.
It does not seem right that Spark has to provide a jar for Hadoop, the lower
level.
>shield yourself by shading is pretty good
For the context, "I" as the user, am not at "fault" here. I am using Spark, and
one dependency component Spark-cassandra. It does not seem right that "I" as
the consumer have to do any shading.
On a separate note, looks like the Guava versioning specifically is a bit
random. Maybe if Spark/Hadoop could not get rid of the dependency, one option
is to copy the code with a different namespace, i.e. a more explicit shading.
My two cents.
> Migrate Guava to 16+?
> -
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 2.0.1
>Reporter: Min Wei
>Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> -val cookie = HashCodes.fromBytes(secret).toString()
> +val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
>} else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
>// A general, soft-reference map for metadata needed during HadoopRDD
> split computation
>// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
>private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
>
> com.google.guava
> guava
> -14.0.1
> -provided
> +19.0
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?
[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394572#comment-15394572
]
Min Wei commented on SPARK-16725:
-
First I am not blocked. Plus I can hack anything I need to make it work given
the OSS nature. Here I am purely curious how Spark plans versioning management
as a more general platform.
As it stands, looks like Spark is "leaking" the Guava dependency. I just did a
web search and looks like there is quite a bit of energy spent on this:
https://groups.google.com/a/lists.datastax.com/forum/#!topic/spark-connector-user/HnTsWJkI5jo
https://issues.apache.org/jira/browse/ZEPPELIN-620
My suggestion is that, the Spark platform needs to provide guidelines. So
spark-cassandra etc. platform pieces built on top should follow it. Otherwise
it will be painful for the upper stack developers or users to consume the whole
stack.
>Spark has to ship a Guava jar because Hadoop needs it
I don't understand this. I assume Hadoop is a dependency for Spark. Spark uses
v14 of Guava to shadows v11 in Hadoop?
>Changing fro 14 to 16 will fix your use case, but what about someone who wants
>a different version?
As long as the version is moving forward, not backwards. Of course in this case
Guava itself could have done a better job of backwards compatibility. [
>"shade your custom dependencies" works for everyone,
Won't this cause code/jar bloat and pain for everyone?
> Migrate Guava to 16+?
> -
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 2.0.1
>Reporter: Min Wei
>Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> -val cookie = HashCodes.fromBytes(secret).toString()
> +val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
>} else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
>// A general, soft-reference map for metadata needed during HadoopRDD
> split computation
>// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
>private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
>
> com.google.guava
> guava
> -14.0.1
> -provided
> +19.0
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16725) Migrate Guava to 16+?
[ https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394311#comment-15394311 ] Min Wei commented on SPARK-16725: - I am not sure if the Hadoop jar is the issue, as I am just using the spark shell for some local testing. I "fixed" my test after upgrading the Guava in the Spark core jar per the diff file. Here is a way to repro it after building the spark-cassandra connector. ./bin/spark-shell --jars ./spark-cassandra-connector-assembly-1.6.0-27-g5760745.jar The following exception goes away after the "upgrade". I agree that there would be other cases where the hadoop jar could cause problems. java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use. This introduces codec resolution issues and potentially other incompatibility issues in the driver. Please upgrade to Guava 16.01 or later. at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62) at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36) at com.datastax.driver.core.Cluster.(Cluster.java:68) at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:37) at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:98) at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:163) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:157) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:157) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:34) at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:60) at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:85) at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:114) at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:127) at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:346) at com.datastax.spark.connector.cql.Schema$.tableFromCassandra(Schema.scala:366) at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:52) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:60) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:60) at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:140) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:60) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:246) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:246) at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1280) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.RDD.take(RDD.scala:1275) at com.datastax.spark.connector.rdd.CassandraRDD.take(CassandraRDD.scala:132) at com.datastax.spark.connector.rdd.CassandraRDD.take(CassandraRDD.scala:133) at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1315) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.RDD.first(RDD.scala:1314) ... 52 elided > Migrate Guava to 16+? > - > > Key: SPARK-16725 > URL: https://issues.apache.org/jira/browse/SPARK-16725 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 2.0.1 >Reporter: Min Wei >Priority: Minor > Original Estimate: 12h > Remaining Estimate: 12h > > Currently Spark depends on an old version of Guava, version 14. However > Spark-cassandra driver asserts on Guava version 16 and above. > It would be great to update the Guava dependency to version 16+ > diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala > b/core/src/main/scala/org/apache/spark/SecurityManager.scala >
[jira] [Commented] (SPARK-16725) Migrate Guava to 16+
[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394084#comment-15394084
]
Min Wei commented on SPARK-16725:
-
Thanks for editing the JIRA fields properly. I used the first/default values in
a haste.
The diff I attached is meant for illustration purposes only. My intent is to
have a discussion of the options. Shading is another option.
Personally I would prefer the upgrade/non-shading option, I assume the Guava
issue won't be the only troublesome jar down the road. Yes, it might open the
can of worms on various versioning management issues. Given the status of Spark
as a general platform, it would be a good problem to solve :-)
My two cents.
> Migrate Guava to 16+
>
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 2.0.1
>Reporter: Min Wei
>Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> -val cookie = HashCodes.fromBytes(secret).toString()
> +val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
>} else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
>// A general, soft-reference map for metadata needed during HadoopRDD
> split computation
>// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
>private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
>
> com.google.guava
> guava
> -14.0.1
> -provided
> +19.0
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16725) Migrate Guava to 16+?
[
https://issues.apache.org/jira/browse/SPARK-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Min Wei updated SPARK-16725:
Summary: Migrate Guava to 16+? (was: Migrate Guava to 16+)
> Migrate Guava to 16+?
> -
>
> Key: SPARK-16725
> URL: https://issues.apache.org/jira/browse/SPARK-16725
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 2.0.1
>Reporter: Min Wei
>Priority: Minor
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> Currently Spark depends on an old version of Guava, version 14. However
> Spark-cassandra driver asserts on Guava version 16 and above.
> It would be great to update the Guava dependency to version 16+
> diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> index f72c7de..abddafe 100644
> --- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
> +++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
> @@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
> import java.security.cert.X509Certificate
> import javax.net.ssl._
>
> -import com.google.common.hash.HashCodes
> +import com.google.common.hash.HashCode
> import com.google.common.io.Files
> import org.apache.hadoop.io.Text
>
> @@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
> val secret = new Array[Byte](length)
> rnd.nextBytes(secret)
>
> -val cookie = HashCodes.fromBytes(secret).toString()
> +val cookie = HashCode.fromBytes(secret).toString()
> SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
> cookie)
> cookie
>} else {
> diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> index af50a6d..02545ae 100644
> --- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
> +++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
> @@ -72,7 +72,7 @@ class SparkEnv (
>
>// A general, soft-reference map for metadata needed during HadoopRDD
> split computation
>// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
> - private[spark] val hadoopJobMetadata = new
> MapMaker().softValues().makeMap[String, Any]()
> + private[spark] val hadoopJobMetadata = new
> MapMaker().weakValues().makeMap[String, Any]()
>
>private[spark] var driverTmpDir: Option[String] = None
>
> diff --git a/pom.xml b/pom.xml
> index d064cb5..7c3e036 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -368,8 +368,7 @@
>
> com.google.guava
> guava
> -14.0.1
> -provided
> +19.0
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16725) Migrate Guava to 16+
Min Wei created SPARK-16725:
---
Summary: Migrate Guava to 16+
Key: SPARK-16725
URL: https://issues.apache.org/jira/browse/SPARK-16725
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 2.0.1
Reporter: Min Wei
Fix For: 2.0.1
Currently Spark depends on an old version of Guava, version 14. However
Spark-cassandra driver asserts on Guava version 16 and above.
It would be great to update the Guava dependency to version 16+
diff --git a/core/src/main/scala/org/apache/spark/SecurityManager.scala
b/core/src/main/scala/org/apache/spark/SecurityManager.scala
index f72c7de..abddafe 100644
--- a/core/src/main/scala/org/apache/spark/SecurityManager.scala
+++ b/core/src/main/scala/org/apache/spark/SecurityManager.scala
@@ -23,7 +23,7 @@ import java.security.{KeyStore, SecureRandom}
import java.security.cert.X509Certificate
import javax.net.ssl._
-import com.google.common.hash.HashCodes
+import com.google.common.hash.HashCode
import com.google.common.io.Files
import org.apache.hadoop.io.Text
@@ -432,7 +432,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf)
val secret = new Array[Byte](length)
rnd.nextBytes(secret)
-val cookie = HashCodes.fromBytes(secret).toString()
+val cookie = HashCode.fromBytes(secret).toString()
SparkHadoopUtil.get.addSecretKeyToUserCredentials(SECRET_LOOKUP_KEY,
cookie)
cookie
} else {
diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
b/core/src/main/scala/org/apache/spark/SparkEnv.scala
index af50a6d..02545ae 100644
--- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
+++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
@@ -72,7 +72,7 @@ class SparkEnv (
// A general, soft-reference map for metadata needed during HadoopRDD split
computation
// (e.g., HadoopFileRDD uses this to cache JobConfs and InputFormats).
- private[spark] val hadoopJobMetadata = new
MapMaker().softValues().makeMap[String, Any]()
+ private[spark] val hadoopJobMetadata = new
MapMaker().weakValues().makeMap[String, Any]()
private[spark] var driverTmpDir: Option[String] = None
diff --git a/pom.xml b/pom.xml
index d064cb5..7c3e036 100644
--- a/pom.xml
+++ b/pom.xml
@@ -368,8 +368,7 @@
com.google.guava
guava
-14.0.1
-provided
+19.0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
