[
https://issues.apache.org/jira/browse/SPARK-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077270#comment-14077270
]
Russell Jurney commented on SPARK-1138:
---------------------------------------
I built spark master with 'sbt/sbt assembly publish-local' and had issues with
my hadoop version, which is CDH 4.4.
Then I built with CDH 4.4, via: 'SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.4.0
sbt/sbt assembly publish-local'. Note: I did not clean. I saw this issue. This
is with Spark trunk when the released version is 1.0.1.
Then I cleaned and rebuilt. The issue persists.
What should I do?
> Spark 0.9.0 does not work with Hadoop / HDFS
> --------------------------------------------
>
> Key: SPARK-1138
> URL: https://issues.apache.org/jira/browse/SPARK-1138
> Project: Spark
> Issue Type: Bug
> Reporter: Sam Abeyratne
>
> UPDATE: This problem is certainly related to trying to use Spark 0.9.0 and
> the latest cloudera Hadoop / HDFS in the same jar. It seems no matter how I
> fiddle with the deps, the do not play nice together.
> I'm getting a java.util.concurrent.TimeoutException when trying to create a
> spark context with 0.9. I cannot, whatever I do, change the timeout. I've
> tried using System.setProperty, the SparkConf mechanism of creating a
> SparkContext and the -D flags when executing my jar. I seem to be able to
> run simple jobs from the spark-shell OK, but my more complicated jobs require
> external libraries so I need to build jars and execute them.
> Some code that causes this:
> println("Creating config")
> val conf = new SparkConf()
> .setMaster(clusterMaster)
> .setAppName("MyApp")
> .setSparkHome(sparkHome)
> .set("spark.akka.askTimeout", parsed.getOrElse(timeouts, "100"))
> .set("spark.akka.timeout", parsed.getOrElse(timeouts, "100"))
> println("Creating sc")
> implicit val sc = new SparkContext(conf)
> The output:
> Creating config
> Creating sc
> log4j:WARN No appenders could be found for logger
> (akka.event.slf4j.Slf4jLogger).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> [ERROR] [02/26/2014 11:05:25.491] [main] [Remoting] Remoting error: [Startup
> timed out] [
> akka.remote.RemoteTransportException: Startup timed out
> at
> akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
> at akka.remote.Remoting.start(Remoting.scala:191)
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> at
> com.adbrain.accuracy.EvaluateAdtruthIDs$.main(EvaluateAdtruthIDs.scala:40)
> at
> com.adbrain.accuracy.EvaluateAdtruthIDs.main(EvaluateAdtruthIDs.scala)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [10000 milliseconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at akka.remote.Remoting.start(Remoting.scala:173)
> ... 11 more
> ]
> Exception in thread "main" java.util.concurrent.TimeoutException: Futures
> timed out after [10000 milliseconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at akka.remote.Remoting.start(Remoting.scala:173)
> at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
> at
> com.adbrain.accuracy.EvaluateAdtruthIDs$.main(EvaluateAdtruthIDs.scala:40)
> at
> com.adbrain.accuracy.EvaluateAdtruthIDs.main(EvaluateAdtruthIDs.scala)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
