Hi Cody, I'm still writing a test to make sure I understood exactly what's going on here, but from looking at the stack trace, it seems like the newer Guava library is picking up the "Optional" class from the Spark assembly.
Could you try one of the options that put the user's classpath before the Spark assembly? (spark.files.userClassPathFirst or spark.yarn.user.classpath.first depending on which master you're running) People seem to have run into issues with those options in the past, but if they work for you, then Guava should pick its own Optional class (instead of the one shipped with Spark) and things should then work. I'll investigate a way to fix it in Spark in the meantime. On Fri, Sep 19, 2014 at 10:30 PM, Cody Koeninger <c...@koeninger.org> wrote: > After the recent spark project changes to guava shading, I'm seeing issues > with the datastax spark cassandra connector (which depends on guava 15.0) > and the datastax cql driver (which depends on guava 16.0.1) > > Building an assembly for a job (with spark marked as provided) that > includes either guava 15.0 or 16.0.1, results in errors like the following: > > scala> session.close > > scala> s[14/09/20 04:56:35 ERROR Futures$CombinedFuture: input future > failed. > java.lang.IllegalAccessError: tried to access class > org.spark-project.guava.common.base.Absent from class > com.google.common.base.Optional > at com.google.common.base.Optional.absent(Optional.java:79) > at com.google.common.base.Optional.fromNullable(Optional.java:94) > at > com.google.common.util.concurrent.Futures$CombinedFuture.setOneValue(Futures.java:1608) > at > com.google.common.util.concurrent.Futures$CombinedFuture.access$400(Futures.java:1470) > at > com.google.common.util.concurrent.Futures$CombinedFuture$2.run(Futures.java:1548) > at > com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101) > at > com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170) > at > com.google.common.util.concurrent.Futures$CombinedFuture.init(Futures.java:1545) > at > com.google.common.util.concurrent.Futures$CombinedFuture.<init>(Futures.java:1491) > at > com.google.common.util.concurrent.Futures.listFuture(Futures.java:1640) > at > com.google.common.util.concurrent.Futures.allAsList(Futures.java:983) > at > com.datastax.driver.core.CloseFuture$Forwarding.<init>(CloseFuture.java:73) > at > com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:398) > at > com.datastax.driver.core.SessionManager.closeAsync(SessionManager.java:157) > at > com.datastax.driver.core.SessionManager.close(SessionManager.java:172) > at > com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$destroySession(CassandraConnector.scala:180) > at > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$5.apply(CassandraConnector.scala:151) > at > com.datastax.spark.connector.cql.CassandraConnector$$anonfun$5.apply(CassandraConnector.scala:151) > at com.datastax.spark.connector.cql.RefCountedCache.com > $datastax$spark$connector$cql$RefCountedCache$$releaseImmediately(RefCountedCache.scala:86) > at > com.datastax.spark.connector.cql.RefCountedCache$ReleaseTask.run(RefCountedCache.scala:26) > at > com.datastax.spark.connector.cql.RefCountedCache$$anonfun$com$datastax$spark$connector$cql$RefCountedCache$$processPendingReleases$2.apply(RefCountedCache.scala:150) > at > com.datastax.spark.connector.cql.RefCountedCache$$anonfun$com$datastax$spark$connector$cql$RefCountedCache$$processPendingReleases$2.apply(RefCountedCache.scala:147) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > scala.collection.concurrent.TrieMapIterator.foreach(TrieMap.scala:922) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.concurrent.TrieMap.foreach(TrieMap.scala:632) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) > at com.datastax.spark.connector.cql.RefCountedCache.com > $datastax$spark$connector$cql$RefCountedCache$$processPendingReleases(RefCountedCache.scala:147) > at > com.datastax.spark.connector.cql.RefCountedCache$$anon$1.run(RefCountedCache.scala:157) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
